A deeper dive into JPA, 2-Phase-Commit [ 2PC ] and RAC

Overview JPA and 2-Phase-Commit

Mike Keith Architect at Oracle and Author

Pro JPA 2: Mastering the Java Persistence API (Second edition)

summarizes the usage of JPA in a distributed evironment the following :

  • A JPA application will get the 2PC benefits the same as any other application
  • The peristence unit data source is using JTA and  is configured to use an XA data source
  • The XA resources and transaction manager 2PC interactions happen on their own without the JPA EMF knowing or having to be involved.
  • If a 2PC XA tx fails then an exception will be thrown just the same as if the tx was optimized to not have 2PC.

This was enough motivation for me working on Oracle RAC and JDBC projects to have a closer look on JPA and 2PC.

Versions used  / Configuration File persistence.xml

Wildfly:  8.2
Hibernate Version: 4.3.7.Final
--> Collecting Data for RAC database1
    Driver Name             : Oracle JDBC driver
    Driver Version          : 12.1.0.2.0
    Database Product Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
    DB Name:  BANKA
    1. Instance Name: bankA_2 - Host: hract21.example.com - Pooled XA Connections: 61

--> Collecting Data for RAC database2
    Driver Name             : Oracle JDBC driver
    Driver Version          : 12.1.0.2.0
    Database Product Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
    DB Name:  BANKB
    1. Instance Name: bankb_3 - Host: hract21.example.com - Pooled XA Connections: 62

persistence.xml

<?xml version="1.0"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"
             version="2.0">

    <persistence-unit name="RacBankAHibPU" transaction-type="JTA">
        <provider>org.hibernate.ejb.HibernatePersistence</provider>
        <jta-data-source>java:/jboss/datasources/xa_rac12g_banka</jta-data-source>
        <class>com.hhu.wfjpa2pc.Accounts</class>
        <properties>
            <property name="hibernate.transaction.jta.platform"
                 value="org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform" />
            <property name="hibernate.show_sql" value="true" />
            <property name="hibernate.dialect" value="org.hibernate.dialect.Oracle10gDialect"/>
        </properties>
    </persistence-unit>
    <persistence-unit name="RacBankBHibPU" transaction-type="JTA">
        <provider>org.hibernate.ejb.HibernatePersistence</provider>
        <jta-data-source>java:/jboss/datasources/xa_rac12g_bankb</jta-data-source>
        <class>com.hhu.wfjpa2pc.Accounts</class>
        <properties>
            <property name="hibernate.transaction.jta.platform"
                 value="org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform" />
            <property name="hibernate.show_sql" value="true" />
            <property name="hibernate.dialect" value="org.hibernate.dialect.Oracle10gDialect"/>
        </properties>
    </persistence-unit>
</persistence>

Running a successful 2PC operation with JPA

Call Flow 

- Get EntityManager for RAC Database1 [ em1=getEntityManager1(); ]
- Get EntityManager for RAC Database2 [ em2=getEntityManager2(); ]
- Start as Usertransacation             [ ut.begin(); ]
- Join transaction from EntityManager 1  [ em1.joinTransaction(); ]
- Join transaction from EntityManager 2  [ em2.joinTransaction(); ]
- Chance Balance on both databases
bankA_acct.setBalance( bankA_acct.getBalance().add(b) );
em1.merge(bankA_acct);
if (isEnableFlush() )
em1.flush();

bankB_acct.setBalance( bankB_acct.getBalance().subtract(b) );
em2.merge(bankB_acct);
if (isEnableFlush() )
em2.flush();
- Finally commit the Transaction [ ut.commit(); ]

Application log :
14:51:58.071 transferMoneyImpl():: Found both Entity Managers for PUs : RacBankAHibPU and RacBankBHibPU
14:51:58.074 transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 10000
14:51:58.075 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 10000
14:51:58.076 transferMoneyImpl():: Both EMs joined our XA Transaction...
14:51:58.092 transferMoneyImpl():: Before Commit ...
14:51:58.160 transferMoneyImpl():: Tx Commit worked !
14:51:58.165 Database Name:BANKA -- Account: User99_at_BANKA -- Balance: 11000.0
14:51:58.168 Database Name:BANKB -- Account: User98_at_BANKB -- Balance: 9000.0
14:51:58.169 transferMoneyImpl():: Leaving with TX Status:: [UT status:  6 - STATUS_NO_TRANSACTION]

-> We successfully managed to transfer some money from bankA to bankB !

Testing Rollback operation with EM flush enabled [ transaction status : STATUS_MARKED_ROLLBACK ]

Account Balance
 transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 20000
 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 0
Note the next money transfer/transaction should trigger a constraint violation ! 

Call Flow
- Get EntityManager for RAC Database1 [ em1=getEntityManager1(); ]
- Get EntityManager for RAC Database2 [ em2=getEntityManager2(); ]
- Start a User transaction             [ ut.begin(); ] 
- Join transaction from EntityManager 1  [ em1.joinTransaction(); ]
- Join transaction from EntityManager 2  [ em2.joinTransaction(); ]
- Chance Balance on both databases
     bankA_acct.setBalance( bankA_acct.getBalance().add(b) );
        em1.merge(bankA_acct);
        if (isEnableFlush() )
          em1.flush();
                
        bankB_acct.setBalance( bankB_acct.getBalance().subtract(b) );
        em2.merge(bankB_acct);           
        if (isEnableFlush() )
          em2.flush();              
- em2.flush is failing due to a constraint violation and set the TX status to  : STATUS_MARKED_ROLLBACK 
   Error : org.hibernate.exception.ConstraintViolationException: could not execute statement
- Exception handler checks transaction status : STATUS_MARKED_ROLLBACK and is rolling back the TX
       if ( status != javax.transaction.Status.STATUS_NO_TRANSACTION   ) 
         {
         ut.rollback();
         ...
- After rollback() transaction status changed to   STATUS_NO_TRANSACTION                      
  
Application log :
15:11:03.920 transferMoneyImpl():: Found both Entity Managers for PUs : RacBankAHibPU and RacBankBHibPU
15:11:03.929 transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 20000
15:11:03.931 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 0
15:11:03.931 transferMoneyImpl():: Both EMs joined our XA Transaction... 
15:11:03.960 transferMoneyImpl():: FATAL ERROR - Tx Status : [UT status:  1 - STATUS_MARKED_ROLLBACK]
15:11:03.962 transferMoneyImpl():: Before TX rollback ... 
15:11:03.974 transferMoneyImpl():: TX rollback worked !
15:11:03.974 transferMoneyImpl():: Leaving with TX Status:: [UT status:  6 - STATUS_NO_TRANSACTION]


Exception stack :
15:11:03.960 Error in top level function: transferMoneyImpl():: 
15:11:03.960 org.hibernate.exception.ConstraintViolationException: could not execute statement
15:11:03.961 javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1683)
    at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1338)
    at com.hhu.wfjpa2pc.Jpa2pcTest.transferMoneyImpl(Jpa2pcTest.java:235)
    at com.hhu.wfjpa2pc.Jpa2pcTest.transferMoney(Jpa2pcTest.java:166)
        ..
Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement
    at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:72)
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java
    ... 
Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-02290: check constraint (SCOTT.S_LOWER_CHK) violated

Testing Rollback operation without EM flush enabled [ transaction status : STATUS_NO_TRANSACTION  ]

Account Balance
 transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 20000
 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 0
Note the next money transfer/transaction should trigger a constraint violation ! 

Call Flow
- Get EntityManager for RAC Database1 [ em1=getEntityManager1(); ]
- Get EntityManager for RAC Database2 [ em2=getEntityManager2(); ]
- Start a User transaction            [ ut.begin(); ] 
- Join transaction from EntityManager 1  [ em1.joinTransaction(); ]
- Join transaction from EntityManager 2  [ em2.joinTransaction(); ]
- Chance Balance on both databases
     bankA_acct.setBalance( bankA_acct.getBalance().add(b) );
        em1.merge(bankA_acct);
        if (isEnableFlush() )
          em1.flush();
                
        bankB_acct.setBalance( bankB_acct.getBalance().subtract(b) );
        em2.merge(bankB_acct);           
        if (isEnableFlush() )
          em2.flush();        
- Commit the Transaction [ ut.commit(); ] fails with :  ARJUNA016053: Could not commit transaction.
- As the Commit itself fails Wildfly rollback the transaction 
- Tx Status after COMMIT error :  STATUS_NO_TRANSACTION 
- Exception handler checks transaction status : STATUS_MARKED_ROLLBACK and is not rolling back the TX
       if ( status != javax.transaction.Status.STATUS_NO_TRANSACTION   ) 
         {
         ut.rollback();
         ...
- Here we don't run any rollback operation -> the TX status remains at   STATUS_NO_TRANSACTION                      
  
Application log :
  15:27:53.818 transferMoneyImpl():: Found both Entity Managers for PUs : RacBankAHibPU and RacBankBHibPU
  15:27:53.827 transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 20000
  15:27:53.829 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 0
  15:27:53.829 transferMoneyImpl():: Both EMs joined our XA Transaction... 
  15:27:53.829 transferMoneyImpl():: Before Commit ... 
  15:27:53.857 transferMoneyImpl():: FATAL ERROR - Tx Status : [UT status:  6 - STATUS_NO_TRANSACTION]
  15:27:53.859 transferMoneyImpl():: TX not active / TX already rolled back
  15:27:53.859 transferMoneyImpl():: Leaving with TX Status:: [UT status:  6 - STATUS_NO_TRANSACTION]

Testing transaction Recovery with JPA

What we are expecting  and what we are testing
  - Transaction Timeout is set to 600 seconds
  - We set a breakpoint at   OracleXAResource.commit
    ==> This means Wildfly has written a COMMIT record to  the  Wildlfly LOG-STORE
  - After stop at the first OracleXAResource.commit breakpoint  we kill the Wildfly server 
  - Both RMs [ Oracle RAC databases ] are now counting down the Transaction Timeout 
  - If Timeout is reached the failed transaction becomes visible in dba_2pc_pending table
  - Trying to get a lock on these records should lead to a ORA-1591 error 
  - After Wildfly restart the Periodic Recovery should run OracleXAResource.commit and release all locks

Preparing and running the test scenario

Start Wildfly in Debug Mode :
Set breakpoint on OracleXAResource.commit and run the application

Stack Trace 
"default task-3"
oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:553)
org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.commit(XAManagedConnection.java:338)
org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:107)
com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:461)
com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2810)
com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2726)
com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1820)
com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1504)
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1166)
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:173)
com.hhu.wfjpa2pc.Jpa2pcTest.transferMoneyImpl(Jpa2pcTest.java:242)
com.hhu.wfjpa2pc.Jpa2pcTest.transferMoney(Jpa2pcTest.java:166)

Wildfly Check for prepared transaction 
$ $WILDFLY_HOME/bin/jboss-cli.sh --connect --file=list_prepared_xa_tx.cli
{"outcome" => "success"}
0:ffffc0a805c9:f5a10ef:56039e68:d

Locate and kill JBOSS server process 
0 S oracle    5875  5821  7  80   0 - 413473 futex_ 08:55 ?       00:00:30 
     /usr/java/latest/bin/java .... -Djboss.server.base.dir=/usr/local/wildfly-8.2.0.Final/standalone -c standalone.xml
0 S oracle    6174  5680  0  80   0 - 25827 pipe_w 09:02 pts/1    00:00:00 grep java
[oracle@wls1 WILDFLY]$ kill -9 5875

Now wait [ at lest 600 seconds ] until the Transaction becomes visible in  dba_2pc_pending

SQL> SELECT * FROM GLOBAL_NAME;
GLOBAL_NAME
----------------
BANKA

SQL> select * from dba_2pc_pending;
LOCAL_TRAN_ID           GLOBAL_TRAN_ID                            STATE         MIX A TRAN_COMMENT
---------------------- ---------------------------------------------------------------- ---------------- --- - ----------------
FAIL_TIM FORCE_TI RETRY_TI OS_USER    OS_TERMINAL  HOST          DB_USER       COMMIT#
-------- -------- -------- ------------ ------------ ---------------- ------------ ----------------
9.21.7139           131077.00000000000000000000FFFFC0A805C90F5A10EF56039E680000000D3 prepared     no
               1
09:07:22      09:15:34 oracle    unknown      wls1.example.com           43619336



SQL> SELECT * FROM GLOBAL_NAME;
GLOBAL_NAME
----------------
BANKB

SQL> select * from dba_2pc_pending;
LOCAL_TRAN_ID           GLOBAL_TRAN_ID                            STATE         MIX A TRAN_COMMENT
---------------------- ---------------------------------------------------------------- ---------------- --- - ----------------
FAIL_TIM FORCE_TI RETRY_TI OS_USER    OS_TERMINAL  HOST          DB_USER       COMMIT#
-------- -------- -------- ------------ ------------ ---------------- ------------ ----------------
4.15.3293           131077.00000000000000000000FFFFC0A805C90F5A10EF56039E680000000D3 prepared     no
               1
09:07:22      09:15:34 oracle    unknown      wls1.example.com           20931538

Check for locks 
-> Connected to  scott/tiger@ract2-scan.grid12c.example.com:1521/banka
select * from accounts for update
*
ERROR at line 1:
ORA-01591: lock held by in-doubt distributed transaction 9.21.7139


-> Connected to  scott/tiger@ract2-scan.grid12c.example.com:1521/bankb
select * from accounts for update
*
ERROR at line 1:
ORA-01591: lock held by in-doubt distributed transaction 4.15.3293


Restart Wildfly in Debug Mode and let the Periodic Recovery Thread commit the transaction 

"Periodic Recovery"
oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:553)
org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.commit(XAManagedConnection.java:338)
org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:107)
com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:461)
com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2810)
com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2726)
com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1820)
com.arjuna.ats.arjuna.recovery.RecoverAtomicAction.replayPhase2(RecoverAtomicAction.java:71)
com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.doRecoverTransaction(AtomicActionRecoveryModule.java:152)
com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.processTransactionsStatus(AtomicActionRecoveryModule.java:253)
com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkSecondPass(AtomicActionRecoveryModule.java:109)
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789)
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)

-> WildFly Thread Periodic Recovery stops at OracleXAResource.commit
-> Press Debugger Command : Continue 
-> WildFly Thread Periodic Recovery has committed Transaction Branch 1
-> WildFly Thread Periodic Recovery stops again  at .OracleXAResource.commit
-> Press Debugger Command : Continue 
-> WildFly Thread Periodic Recovery has committed Transaction Branch 2
-> Complete Transaction is now committed 

Verify Access to the Database records and Wildfly Prepared Transaction Cleanup 
-> Connected to  scott/tiger@ract2-scan.grid12c.example.com:1521/banka
ACCOUNT                 BALANCE
-------------------------------- ----------
User99_at_BANKA               14000


-> Connected to  scott/tiger@ract2-scan.grid12c.example.com:1521/bankb
ACCOUNT                 BALANCE
-------------------------------- ----------
User98_at_BANKB                6000


List prepared Transaction
$  $WILDFLY_HOME/bin/jboss-cli.sh --connect --file=list_prepared_xa_tx.cli
{"outcome" => "success"}

-> After a successful transaction recovery the locks are gone 

 

Java Code

public void transferMoneyImpl()
      {
        String methodName = "transferMoneyImpl():: ";
        EntityManager em1;
        EntityManager em2;
        UserTransaction ut =null;
      try
        {
        setRunTimeInfo(methodName  + "Entering ... ");
            
        HttpSession session = (HttpSession) FacesContext.getCurrentInstance().getExternalContext().getSession(true);
        if ( session == null)
              {
                throw new IllegalArgumentException(methodName+ ": Could not get HTTP session : ");    
              }                        
        final Object lock = session.getId().intern();       
        synchronized(lock) 
              {
                em1=getEntityManager1();
                em2=getEntityManager2();
                    //
                    // Note even we get an EntityManager Object we still not sure that the 
                    // EntityManager Could open connection the underlying JDBC connection !
                    //
                if ( em1 == null )
                    setRunTimeInfo(methodName  + "Faild to get EM for PU: " + EMF.getPU1() );
                else if ( em2 == null )
                    setRunTimeInfo(methodName  + "Faild to get EM for PU: " + EMF.getPU2() );
                else
                    setRunTimeInfo(methodName  + "Found both Entity Managers for PUs : " + 
                       EMF.getPU1()  + " and " +  EMF.getPU2()  ); 
                   
                 
                String bankA_acct_name = "User99_at_BANKA";
                Accounts bankA_acct = em1.find(Accounts.class, bankA_acct_name);
                if ( bankA_acct == null)
                    { 
                    setRunTimeInfo(methodName + "Could not locate Account at bankA : " + bankA_acct_name );
                    return;
                    }
                setRunTimeInfo(methodName  +"Account at bank A: " + bankA_acct.getAccount()  + " - Balance: " +  bankA_acct.getBalance() );
                
                String bankB_acct_name = "User98_at_BANKB";
                Accounts bankB_acct = em2.find(Accounts.class, bankB_acct_name);
                if ( bankB_acct == null)
                    { 
                    setRunTimeInfo(methodName + "Could not locate Account at bankB : " + bankB_acct_name );
                    return;
                    }
                setRunTimeInfo(methodName  +"Account at bank B: " + bankB_acct.getAccount()  + " - Balance: " +  bankB_acct.getBalance() );
              
                ut  = (javax.transaction.UserTransaction)new InitialContext().lookup("java:comp/UserTransaction"); 
                    // Set tranaction time to 120 seconds to avoid any timeouts during testing -
                    // especially when testing transaction recovery by restarting Wildfly server 
                    // Note as we kill the JAVA process both RMs will wait 120 s before Tx becomes visible in dba_2pc_pending 
                int tx_timeout = 120;
                ut.setTransactionTimeout(tx_timeout);
                ut.begin();
                em1.joinTransaction();
                em2.joinTransaction();
                setRunTimeInfo(methodName  + "Both EMs joined our XA Transaction... - TX Timeout: " + tx_timeout );
                BigDecimal b = new BigDecimal(1000);
                bankA_acct.setBalance( bankA_acct.getBalance().add(b) );
                em1.merge(bankA_acct);
                if (isEnableFlush() )
                    em1.flush();
                
                bankB_acct.setBalance( bankB_acct.getBalance().subtract(b) );
                em2.merge(bankB_acct);           
                if (isEnableFlush() )
                    em2.flush();
                
                setRunTimeInfo(methodName  + "Before Commit ... ");                
                ut.commit();
                setRunTimeInfo(methodName  + "Tx Commit worked !");
                checkBalanceImpl();
              }
        } catch ( Throwable t1)
          { 
            try
              {    
              String tx_status = returnTXStatus(ut);
              setRunTimeInfo( methodName  + "FATAL ERROR - Tx Status : " + tx_status  );
                // Use Throwable as we don't want to loose any important imformation
                // Note: Throwable is super class of Exception class          
               genericException("Error in top level function: " + methodName , (Exception)t1);                          
               if ( ut != null )
                  {
                    int status = ut.getStatus();    
                        // rollback transaction if still active - if not do nothing 
                    if ( status != javax.transaction.Status.STATUS_NO_TRANSACTION   ) {
                        setRunTimeInfo(methodName  + "Before TX rollback ... ");
                        ut.rollback();
                        setRunTimeInfo(methodName  + "TX rollback worked !");
                    } else
                        setRunTimeInfo(methodName  + "TX not active / TX already rolled back");
                  }
              }  catch ( Throwable t2)
                 { 
                   genericException(methodName + "FATAL ERROR during ut.rollback() ", (Exception)t2); 
                 } 
          }
        closeEntityManagers();       
        String tx_status_exit = "";
        try
          {    
            tx_status_exit = returnTXStatus(ut);
          }   catch ( Throwable t3)
            { 
              genericException(methodName + " Error during returning TX status ", (Exception)t3); 
            }    
        setRunTimeInfo(methodName  + "Leaving with TX Status:: " + tx_status_exit );
      }

Reference

One thought on “A deeper dive into JPA, 2-Phase-Commit [ 2PC ] and RAC”

Leave a Reply

Your email address will not be published. Required fields are marked *