Thursday, January 22, 2026

Installing Oracle AI Database 26ai (23.26.1) on CentOS Stream 9

 A comprehensive guide for installing Oracle AI Database 26ai Enterprise Edition on EC2 instances running CentOS Stream 9.

Overview

This guide covers the installation of Oracle AI Database 26ai (Release Update 23.26.1.0.0) using the Gold Image distribution. The installation uses silent mode for automation and creates a container database (CDB) with a pluggable database (PDB).

Version Information:

  • Oracle AI Database 26ai Enterprise Edition
  • Release: 23.26.1.0.0
  • Patch: 38743961 (Gold Image)
  • Platform: Linux x86-64

Prerequisites

System Requirements

Minimum Hardware:

  • CPU: 2+ cores recommended
  • RAM: 4 GB minimum (8 GB+ recommended for production)
  • Storage: 20 GB minimum for software and database files
  • Swap: 2 GB minimum

Supported Operating Systems:

  • Oracle Linux 9.2+ (UEK7 or RHCK)
  • Oracle Linux 8.6+ (UEK6/UEK7 or RHCK)
  • Red Hat Enterprise Linux 9.2+
  • CentOS Stream 9 (working, but NOT certified)
  • SUSE Linux Enterprise Server 15 SP5

Verify Your System:

cat /etc/redhat-release
uname -r

Download Oracle Software

  1. Navigate to Oracle Support
  2. Search for Patch 38743961 - Database Release Update 23.26.1.0.0 Gold Image
  3. Download: p38743961_230000_Linux-x86-64.zip (2.2 GB)

Installation Steps

Step 1: Install Required Packages

Install all prerequisite packages required by Oracle Database:

sudo dnf install -y \
  bc \
  binutils \
  compat-openssl11 \
  elfutils-libelf \
  fontconfig \
  glibc \
  glibc-devel \
  glibc-headers \
  ksh \
  libaio \
  libasan \
  liblsan \
  libX11 \
  libXau \
  libXi \
  libXrender \
  libXtst \
  libxcrypt-compat \
  libgcc \
  libibverbs \
  librdmacm \
  libstdc++ \
  libxcb \
  libvirt-libs \
  make \
  policycoreutils \
  policycoreutils-python-utils \
  smartmontools \
  sysstat

Note: For Oracle Linux users, consider using the Oracle AI Database Preinstallation RPM which automatically configures the system and installs required packages.

Step 2: Create Oracle Groups

Create the required Oracle Inventory and database administration groups:

# Primary installation group
sudo /usr/sbin/groupadd -g 54321 oinstall

# Database administration groups
sudo /usr/sbin/groupadd -g 54322 dba
sudo /usr/sbin/groupadd -g 54323 oper
sudo /usr/sbin/groupadd -g 54324 backupdba
sudo /usr/sbin/groupadd -g 54325 dgdba
sudo /usr/sbin/groupadd -g 54326 kmdba
sudo /usr/sbin/groupadd -g 54330 racdba

Group Purposes:

  • oinstall: Oracle Inventory group (primary)
  • dba: Database Administrator (SYSDBA)
  • oper: Database Operator (SYSOPER)
  • backupdba: Backup and Recovery (SYSBACKUP)
  • dgdba: Data Guard (SYSDG)
  • kmdba: Encryption Key Management (SYSKM)
  • racdba: Real Application Clusters

Step 3: Create Oracle User

Create the oracle user for database installation:

sudo /usr/sbin/useradd -u 54321 -g oinstall \
  -G dba,backupdba,dgdba,kmdba,racdba oracle

(Optional) Set a password for the oracle user:

sudo passwd oracle

Step 4: Configure Resource Limits

Configure kernel resource limits for the oracle user:

sudo tee -a /etc/security/limits.conf << 'EOF'
# Oracle Database resource limits
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle soft memlock 28835840
oracle hard memlock 28835840
EOF

Limit Explanations:

  • nofile: Maximum number of open file descriptors
  • nproc: Maximum number of processes
  • stack: Maximum stack size (KB)
  • memlock: Maximum locked memory (KB)

Step 5: Create Directory Structure

Create the Oracle base, inventory, and staging directories:

# Create directories as root
sudo mkdir -p /u01/app/oracle
sudo mkdir -p /u01/app/oraInventory
sudo mkdir -p /u01/stage

# Set ownership
sudo chown -R oracle:oinstall /u01/app/oracle
sudo chown -R oracle:oinstall /u01/app/oraInventory
sudo chown oracle:oinstall /u01/stage

# Set permissions
sudo chmod -R 775 /u01/app

Directory Structure:

  • /u01/app/oracle: Oracle Base directory
  • /u01/app/oraInventory: Oracle Inventory location
  • /u01/stage: Staging area for installation files

Step 6: Upload and Extract Installation Files

Transfer the downloaded zip file to the staging directory and extract:

# Switch to oracle user
sudo su - oracle

# Create Oracle Home directory
mkdir -p /u01/app/oracle/product/23.0.0/dbhome_1

# Navigate to Oracle Home
cd /u01/app/oracle/product/23.0.0/dbhome_1

# Extract the Gold Image (assumes zip is in /u01/stage)
unzip -q /u01/stage/p38743961_230000_Linux-x86-64.zip

Important: The Gold Image contains both the Oracle Database software and the Release Update pre-applied.

Step 7: Run Silent Installation

Execute the Oracle installer in silent mode to install software and create the database:

./runInstaller -silent -createDatabase \
  -OSDBA dba \
  -OSBACKUPDBA backupdba \
  -OSDGDBA dgdba \
  -OSKMDBA kmdba \
  -OSRACDBA racdba \
  -OSOPER oper \
  -installEdition EE \
  -ORACLE_BASE /u01/app/oracle \
  -dataLocation /u01/app/oracle/oradata \
  -gdbName orcl23.example.com \
  -pdbName pdb1 \
  -dbSID orcl23 \
  -memoryLimit 4000 \
  -INVENTORY_LOCATION /u01/app/oraInventory \
  -useSamePasswordForAllSchemas \
  -ignorePrereq

Installation Parameters:

  • -createDatabase: Creates database during installation
  • -installEdition EE: Enterprise Edition
  • -gdbName: Global database name (CDB)
  • -pdbName: Pluggable database name
  • -dbSID: System identifier
  • -memoryLimit: Memory in MB (4000 = 4 GB)
  • -useSamePasswordForAllSchemas: Simplifies password management
  • -ignorePrereq: Bypasses prerequisite checks (use cautiously)
  • -executeConfigTools: Runs NETCA and DBCA automatically (The -executeConfigTools flag can only be used for an Oracle home software that has been already installed using the configure or upgrade options. Ensure that the orainstRoot.sh script, from the inventory location, has been run.)

You'll be prompted to enter passwords for database schemas (SYS, SYSTEM, PDBADMIN).

Step 8: Execute Root Scripts

After installation completes, run the required root scripts as indicated by the installer:

# Exit oracle user session
exit

# Run as root
sudo /u01/app/oraInventory/orainstRoot.sh
sudo /u01/app/oracle/product/23.0.0/dbhome_1/root.sh

These scripts configure inventory permissions and set up Oracle Restart components.

Step 9: Configure Network and Database (If Not Auto-Configured)

If you didn't use -executeConfigTools, run the configuration manually:

# Switch back to oracle user
sudo su - oracle

# Navigate to Oracle Home
cd /u01/app/oracle/product/23.0.0/dbhome_1

# Run configuration tools
./runInstaller -silent \
  -useSamePasswordForAllSchemas \
  -executeConfigTools

This will:

  1. Run NETCA (Network Configuration Assistant) to create the listener
  2. Run DBCA (Database Configuration Assistant) to create the database

Expected Output:

  • NETCA creates listener on port 1521
  • DBCA creates CDB orcl23 with PDB pdb1
  • Database creation takes 3-5 minutes

Post-Installation Configuration

Step 10: Set Environment Variables

Add Oracle environment variables to the oracle user's profile:

# Switch to oracle user
sudo su - oracle

# Edit .bash_profile
cat >> ~/.bash_profile << 'EOF'

# Oracle Environment
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/23.0.0/dbhome_1
export ORACLE_SID=orcl23
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
EOF

# Apply changes
source ~/.bash_profile

Step 11: Verify Installation

Check the installed patches and database version:

# Verify patches
$ORACLE_HOME/OPatch/opatch lspatches

Expected Output:

38743688;OCW RELEASE UPDATE 23.26.1.0.0 (GOLD IMAGE)
38743669;Database Release Update : 23.26.1.0.0
OPatch succeeded.

Step 12: Test Database Connectivity

Connect to the database using SQLcl or SQL*Plus:

# Using SQLcl (if installed)
sql system@localhost/pdb1.example.com

# Or using SQL*Plus
sqlplus system@localhost/pdb1.example.com

Verify Version:

SELECT banner FROM v$version;

Expected Output:

Oracle AI Database 26ai Enterprise Edition Release 23.26.1.0.0 - Production

Step 13: Configure Automatic Startup (Optional)

Edit /etc/oratab to enable automatic startup:

sudo vi /etc/oratab

Change the last field from N to Y:

orcl23:/u01/app/oracle/product/23.0.0/dbhome_1:Y

Create a systemd service for automatic startup:

sudo tee /etc/systemd/system/oracle-database-orcl23.service << 'EOF'
[Unit]
Description=Oracle Database Service
After=network.target

[Service]
Type=forking
User=oracle
Group=oinstall
Environment="ORACLE_HOME=/u01/app/oracle/product/23.0.0/dbhome_1"
Environment="ORACLE_SID=orcl23"
Environment="PATH=/u01/app/oracle/product/23.0.0/dbhome_1/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
Environment="LD_LIBRARY_PATH=/u01/app/oracle/product/23.0.0/dbhome_1/lib"
ExecStart=/bin/bash /u01/app/oracle/product/23.0.0/dbhome_1/bin/dbstart /u01/app/oracle/product/23.0.0/dbhome_1
ExecStop=/bin/bash /u01/app/oracle/product/23.0.0/dbhome_1/bin/dbshut /u01/app/oracle/product/23.0.0/dbhome_1
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

# Enable the service
sudo systemctl daemon-reload
sudo systemctl enable oracle-database-orcl23.service

Database Management

Starting and Stopping the Database

Manual Start:

# Start listener
lsnrctl start

# Start database
sqlplus / as sysdba
SQL> STARTUP;
SQL> ALTER PLUGGABLE DATABASE ALL OPEN;
SQL> EXIT;

Manual Stop:

sqlplus / as sysdba
SQL> SHUTDOWN IMMEDIATE;
SQL> EXIT;

# Stop listener
lsnrctl stop

Using dbstart/dbshut:

# Start
dbstart $ORACLE_HOME

# Stop
dbshut $ORACLE_HOME

Connecting to Databases

Connect to CDB:

sqlplus / as sysdba
# or
sqlplus sys@localhost/orcl23.example.com as sysdba

Connect to PDB:

sqlplus system@localhost/pdb1.example.com

Switch Between Containers:

-- Show current container
SHOW CON_NAME;

-- Switch to PDB
ALTER SESSION SET CONTAINER = pdb1;

-- Switch to CDB root
ALTER SESSION SET CONTAINER = CDB$ROOT;

Troubleshooting

Common Issues

Issue: Installation fails with prerequisite check errors

  • Solution: Review /u01/app/oraInventory/logs/InstallActions*.log
  • Consider using -ignorePrereq flag (not recommended for production)

Issue: Listener not starting

# Check listener status
lsnrctl status

# Check listener.ora
cat $ORACLE_HOME/network/admin/listener.ora

# Restart listener
lsnrctl stop
lsnrctl start

Issue: Database not accessible

# Check database status
ps -ef | grep pmon

# Check alert log
tail -f $ORACLE_BASE/diag/rdbms/orcl23/orcl23/trace/alert_orcl23.log

Issue: Memory allocation errors

  • Increase system memory or reduce -memoryLimit parameter
  • Check available memory: free -h

Log Locations

  • Installation logs: /u01/app/oraInventory/logs/
  • DBCA logs: /u01/app/oracle/cfgtoollogs/dbca/orcl23/
  • Alert log: $ORACLE_BASE/diag/rdbms/orcl23/orcl23/trace/alert_orcl23.log
  • Listener log: $ORACLE_BASE/diag/tnslsnr/$HOSTNAME/listener/trace/listener.log

Security Considerations

  1. Change default passwords immediately for SYS, SYSTEM, and PDBADMIN users
  2. Configure firewall rules to restrict database port access (1521)
  3. Enable SSL/TLS for encrypted connections in production
  4. Regular patching: Subscribe to Oracle Critical Patch Updates (CPU)
  5. Audit configuration: Enable database auditing for compliance
  6. Backup strategy: Implement RMAN backup procedures

Storage Configuration

# Example: Mount additional EBS volume for data files
sudo mkfs.xfs /dev/nvme1n1
sudo mkdir -p /u02/oradata
sudo mount /dev/nvme1n1 /u02/oradata
sudo chown -R oracle:oinstall /u02/oradata

# Add to /etc/fstab for persistence
echo "/dev/nvme1n1 /u02/oradata xfs defaults,nofail 0 2" | sudo tee -a /etc/fstab

References

Saturday, December 27, 2025

Downloading Oracle Patches with AutoUpgrade

 Reference: The Easiest Way to Download 19.27 Release Update

Prerequisites

  • AutoUpgrade JAR file (autoupgrade.jar)
  • Java 11 (AutoUpgrade Patching requires Java 11, not Java 21)
  • My Oracle Support (MOS) account credentials

Steps

1. Switch to Java 11

AutoUpgrade Patching requires Java 11:

export JAVA_HOME=$(/usr/libexec/java_home -v 11)

To make this permanent, add to ~/.zprofile:

vi ~/.zprofile
# Add the export command above

2. Create AutoUpgrade Configuration File

Create autoupgrade_keystore.cfg with your patch requirements:

vi autoupgrade_keystore.cfg

global.global_log_dir=/Users/donghua/logs
global.keystore=/Users/donghua/autoupgrade_keystore
global.folder=/Users/donghua/Software_Repo/Oracle/19.29
patch1.platform=LINUX.X64
patch1.patch=RU:19.29,OPATCH,OJVM

3. Set Up MOS Credentials

Initialize the keystore and store your MOS credentials:

java -jar autoupgrade.jar -config autoupgrade_keystore.cfg -patch -load_password

Follow the prompts:

  1. Create keystore password (enter twice)
  2. Type add -user <your-email@youremail.com>
  3. Enter your MOS password (twice)
  4. Type save
  5. Convert to auto-login: YES
  6. Type exit

4. Create Download Directory

mkdir ~/Software_Repo/Oracle/19.29

5. Download Patches

java -jar autoupgrade.jar -config autoupgrade_keystore.cfg -patch -mode download

This will:

  • Connect to MOS
  • Search for specified patches
  • Download to the configured directory
  • Validate downloaded files

Example Output

Downloaded patches for 19.29:

  • p38291812_190000_Linux-x86-64.zip - DATABASE RELEASE UPDATE 19.29.0.0.0
  • p38194382_190000_Linux-x86-64.zip - OJVM RELEASE UPDATE 19.29.0.0.0
  • p6880880_190000_Linux-x86-64.zip - OPatch 12.2.0.1.48

All files are validated after download.

Tuesday, November 25, 2025

MultiXact Contention: Source Code Analysis

 

The Mystery

Observation:

  • INSERT-only (400 connections): Massive MultiXact SLRU activity (75K hits/sec), but 0% LWLock wait events
  • INSERT+UPDATE (500 connections): Similar SLRU activity, but 7% MultiXact LWLock wait events

Question: Why does adding UPDATE cause MultiXact LWLock contention to become visible?

Source Code Evidence

1. When MultiXacts Are Created

File: src/backend/access/heap/heapam.c:5476-5556

else if (TransactionIdIsInProgress(xmax))
{
    /*
     * If the XMAX is a valid, in-progress TransactionId, then we need to
     * create a new MultiXactId that includes both the old locker or
     * updater and our own TransactionId.
     */
    MultiXactStatus new_status;
    MultiXactStatus old_status;
    
    // Determine old lock type
    if (HEAP_XMAX_IS_LOCKED_ONLY(old_infomask))
    {
        if (HEAP_XMAX_IS_KEYSHR_LOCKED(old_infomask))
            old_status = MultiXactStatusForKeyShare;
        // ... other lock types
    }
    
    // Create new MultiXact with both transactions
    new_status = get_mxact_status_for_lock(mode, is_update);
    new_xmax = MultiXactIdCreate(xmax, old_status,
                                 add_to_xmax, new_status);
}

Key insight: MultiXact is created when a transaction encounters an in-progress xmax on a tuple.

2. MultiXact Expansion Requires SLRU Read

File: src/backend/access/transam/multixact.c:478-560

MultiXactId
MultiXactIdExpand(MultiXactId multi, TransactionId xid, MultiXactStatus status)
{
    MultiXactMember *members;
    int nmembers;
    
    // MUST READ existing MultiXact members from SLRU
    nmembers = GetMultiXactIdMembers(multi, &members, false, false);
    
    // Check if already a member (optimization)
    for (i = 0; i < nmembers; i++)
    {
        if (TransactionIdEquals(members[i].xid, xid) &&
            (members[i].status == status))
        {
            // Already member, return existing MultiXact
            return multi;
        }
    }
    
    // Filter out dead members
    for (i = 0, j = 0; i < nmembers; i++)
    {
        if (TransactionIdIsInProgress(members[i].xid) ||
            (ISUPDATE_from_mxstatus(members[i].status) &&
             TransactionIdDidCommit(members[i].xid)))
        {
            newMembers[j++] = members[i];
        }
    }
    
    // Add new member and create new MultiXact
    newMembers[j].xid = xid;
    newMembers[j++].status = status;
    newMulti = MultiXactIdCreateFromMembers(j, newMembers);
    
    return newMulti;
}

Key insight: Every MultiXact expansion requires:

  1. Reading existing members from SLRU (GetMultiXactIdMembers)
  2. Filtering dead members
  3. Creating new MultiXact with updated member list

3. GetMultiXactIdMembers Accesses SLRU

File: src/backend/access/transam/multixact.c:1290-1400

GetMultiXactIdMembers(MultiXactId multi, MultiXactMember **members, ...)
{
    // Check local cache first
    length = mXactCacheGetById(multi, members);
    if (length >= 0)
        return length;  // Cache hit - no SLRU access
    
    // Cache miss - must read from SLRU
    // Acquire LWLock on MultiXactGenLock
    LWLockAcquire(MultiXactGenLock, LW_SHARED);
    oldestMXact = MultiXactState->oldestMultiXactId;
    nextMXact = MultiXactState->nextMXact;
    LWLockRelease(MultiXactGenLock);
    
    // Read offset from MultiXactOffset SLRU
    // Then read members from MultiXactMember SLRU
    // (code continues with SLRU page access requiring LWLocks)
}

Key insight: SLRU access requires acquiring LWLocks on:

  • MultiXactOffsetBuffer - for offset SLRU buffer
  • MultiXactMemberBuffer - for member SLRU buffer
  • MultiXactOffsetSLRU - for offset SLRU I/O
  • MultiXactMemberSLRU - for member SLRU I/O

Why INSERT-Only Shows No LWLock Contention

Scenario: 400 INSERT connections on 2 parent rows

What happens:

  1. INSERT 1: Parent xmax = 0

    • Acquires KEY SHARE
    • Sets xmax = Txn1 (single transaction ID)
    • Commits quickly (milliseconds)
  2. INSERT 2: Parent xmax = Txn1 (committed)

    • TransactionIdIsInProgress(Txn1) = FALSE (already committed)
    • Falls through to "no locker" case (line 5352)
    • Sets xmax = Txn2 (single transaction ID)
    • No MultiXact created!
  3. INSERT 3, 4, 5...: Same pattern

    • Previous transaction already committed
    • Just replace xmax with own transaction ID
    • Fast, no MultiXact operations

Even when MultiXacts ARE created:

  • All transactions use KEY SHARE (same lock type)
  • Compatible locks - no blocking
  • Transactions complete in milliseconds
  • LWLock acquisition is fast and non-contended
  • No visible wait events

Why UPDATE+INSERT Shows LWLock Contention

Scenario: 100 UPDATE + 400 INSERT connections on 2 parent rows

What happens:

  1. UPDATE 1: Acquires FOR UPDATE on parent

    • Sets xmax = UpdateTxn1
    • Transaction in-progress (even without pg_sleep, takes milliseconds)
  2. INSERT 1: Needs KEY SHARE on same parent

    • Sees xmax = UpdateTxn1
    • TransactionIdIsInProgress(UpdateTxn1) = TRUE
    • Must create MultiXact (line 5553):
      new_xmax = MultiXactIdCreate(UpdateTxn1, MultiXactStatusUpdate,
                                   InsertTxn1, MultiXactStatusForKeyShare);
      
    • Parent xmax = MultiXact1
  3. INSERT 2: Needs KEY SHARE on same parent

    • Sees xmax = MultiXact1
    • Must expand MultiXact (line 5398):
      new_xmax = MultiXactIdExpand(MultiXact1, InsertTxn2, 
                                   MultiXactStatusForKeyShare);
      
    • Calls GetMultiXactIdMembers() → SLRU read
    • Creates MultiXact2 with 3 members
  4. UPDATE 2: Tries to update same parent

    • Sees xmax = MultiXact2 (contains UpdateTxn1)
    • BLOCKS on Lock:tuple waiting for UpdateTxn1
    • Transaction queued
  5. INSERT 3, 4, 5... (400 connections):

    • All see MultiXact xmax
    • All must call GetMultiXactIdMembers() → SLRU reads
    • All must expand MultiXact → SLRU writes
  6. UPDATE 3, 4, 5... (100 connections):

    • All queued waiting for lock
    • All holding or waiting for LWLocks on MultiXact structures

The Contention Cascade

With 100 UPDATEs queued + 400 INSERTs running:

  • Same 2 parent rows = hot spot
  • Mixed lock types (FOR UPDATE + KEY SHARE) = complex MultiXact operations
  • Sustained in-progress transactions = constant MultiXact expansion
  • 500 concurrent connections all accessing same MultiXact SLRU pages
  • LWLock contention on:
    • MultiXactOffsetBuffer (10%)
    • MultiXactMemberBuffer (6%)
    • MultiXactOffsetSLRU (1%)
    • MultiXactMemberSLRU (1%)

Proof from Testing

Test 1: Session overlap creates MultiXact

-- Session 1: UPDATE in progress
BEGIN;
UPDATE locations SET loc_name = 'junk-001' WHERE loc_id = 1;
-- xmax = 1567133 (in-progress)

-- Session 2: INSERT while UPDATE in-progress
INSERT INTO users (loc_id, fname) VALUES (1, 'test');
-- Parent xmax changes: 1567133 → 1897574 (MultiXact created!)

Test 2: SLRU activity proves MultiXact operations

INSERT-only workload:

MultiXactMember blks_hit: 19,932,415 → 20,073,773 (75K hits/sec)
MultiXactOffset blks_hit: 19,872,893 → 20,014,799 (75K hits/sec)
LWLock wait events: 0%

INSERT+UPDATE workload:

MultiXactMember blks_hit: 17,198,033 → 17,228,164 (similar rate)
MultiXactOffset blks_hit: 17,148,469 → 17,178,119 (similar rate)
LWLock wait events: 7% (MultiXact-related)

Conclusion

Why INSERT-only doesn't show LWLock contention:

  1. Transactions commit quickly (milliseconds)
  2. Next transaction sees committed xmax → no MultiXact needed (optimization)
  3. Even when MultiXacts created, all KEY SHARE locks are compatible
  4. No blocking → LWLocks acquired/released quickly
  5. No visible wait events

Why UPDATE+INSERT shows LWLock contention:

  1. UPDATE creates in-progress xmax with FOR UPDATE lock
  2. INSERT must create/expand MultiXact to track mixed lock types
  3. UPDATE-UPDATE blocking creates transaction queue (Lock:tuple 11%)
  4. 500 connections all accessing same MultiXact SLRU pages for 2 hot rows
  5. Sustained pressure on MultiXact infrastructure
  6. LWLock contention becomes visible as wait events (7%)

The root cause: High concurrency + hot rows + mixed lock types + transaction queuing = MultiXact LWLock bottleneck.

References