TSQLFile
Access an SQL db via the TFile interface.
The main motivation for the TSQLFile development is to have
"transparent" access to SQL data base via standard TFile interface.
The main approach that each class (but not each object) has one or two tables
with names like $(CLASSNAME)_ver$(VERSION) and $(CLASSNAME)_raw$(VERSION)
For example: TAxis_ver8 or TList_raw5
Second kind of tables appears, when some of class members can not be converted to
normalized form or when class has custom streamer.
For instance, for TH1 class two tables are required: TH1_ver4 and TH1_raw4
Most of members are stored in TH1_ver4 table column-wise, and only member:
Double_t* fBuffer; //[fBufferSize]
can not be represented as column while size of array is not fixed.
Therefore, fBuffer will be written as list of values in TH1_raw4 table.
All objects, stored in the DB, will be registered in table "ObjectsTable".
In this there are following columns:
| Name | Description |
|------|-------------|
| "key:id" | key identifier to which belong object |
| "obj:id" | object identifier |
| "Class" | object class name |
| "Version" | object class version |
Data in each "ObjectsTable" row uniquely identify, in which table
and which column object is stored.
In normal situation all class data should be sorted column-wise.
Up to now following member are supported:
-# Basic data types. Here is everything clear. Column SQL type will be as much as possible
close to the original type of value.
-# Fixed array of basic data types. In this case n columns like fArr[0],
fArr[1] and so on will be created.
If there is multidimensional array, names will be fArr2[1][2][1] and so on
-# Parent class. In this case version of parent class is stored and
data of parent class will be stored with the same obj:id in correspondent table.
There is a special case, when parent store nothing (this is for instance TQObject).
In that case just -1 is written to avoid any extra checks if table exist or not.
-# Object as data member. In that case object is saved in normal way to data base and column
will contain id of this object.
-# Pointer on object. Same as before. In case if object was already stored, just its id
will be placed in the column. For NULL pointer 0 is used.
-# TString. Now column with limited width like VARCAHR(255) in MySQL is used.
Later this will be improved to support maximum possible strings
-# Anything else. Data will be converted to raw format and saved in _streamer_ table.
Each row supplied with obj:id and row:id, where row:id indicates
data, corresponding to this particular data member, and column
will contain this raw:id
All conversion to SQL statements are done with help of TSQLStructure class.
This is special hierarchical structure wich internally is very similar
to XML structures. TBufferSQL2 creates these structures, when object
data is streamed by ROOT and only afterwards all SQL statements will be produced
and applied all together.
When data is reading, TBufferSQL2 will produce requests to database
during unstreaming of object data.
Optionally (default this options on) name of column includes
suffix which indicates type of column. For instance:
| Name | Description |
|------|-------------|
| *:parent | parent class, column contain class version |
| *:object | other object, column contain object id |
| *:rawdata | raw data, column contains id of raw data from _streamer_ table |
| *:Int_t | column with integer value |
Use TSQLFile::SetUseSuffixes(kFALSE) to disable suffixes usage.
This and several other options can be changed only when
TSQLFile created with options "CREATE" or "RECREATE" and only before
first write operation. These options are:
| Name | Description |
|------|-------------|
| SetUseSuffixes() | suffix usage in column names (default - on) |
| SetArrayLimit() | defines maximum array size, which can has column for each element (default 21) |
| SetTablesType() | table type name in MySQL database (default "InnoDB") |
| SetUseIndexes() | usage of indexes in database (default kIndexesBasic) |
Normally these functions should be called immediately after TSQLFile constructor.
When objects data written to database, by default START TRANSACTION/COMMIT
SQL commands are used before and after data storage. If TSQLFile detects
any problems, ROLLBACK command will be used to restore
previous state of data base. If transactions not supported by SQL server,
they can be disabled by SetUseTransactions(kTransactionsOff). Or user
can take responsibility to use transactions function himself.
By default only indexes for basic tables are created.
In most cases usage of indexes increase performance to data reading,
but it also can increase time of writing data to database.
There are several modes of index usage available in SetUseIndexes() method
There is MakeSelectQuery(TClass*) method, which
produces SELECT statement to get objects data of specified class.
Difference from simple statement like:
mysql> SELECT * FROM TH1I_ver1
that not only data for that class, but also data from parent classes
will be extracted from other tables and combined in single result table.
Such select query can be useful for external access to objects data.
Up to now MySQL 4.1 and Oracle 9i were tested.
Some extra work is required for other SQL databases.
Hopefully, this should be straightforward.
Known problems and open questions.
-# TTree is not supported by TSQLFile. There is independent development
of TTreeSQL class, which allows to store trees directly in SQL database
-# TClonesArray is store objects in raw format,
which can not be accessed outside ROOT.
This will be changed later.
-# TDirectory cannot work. Hopefully, will (changes in ROOT basic I/O is required)
-# Streamer infos are not written to file, therefore schema evolution
is not yet supported. All eforts are done to enable this feature in
the near future
- "READ / OPEN"
- "UPDATE"
- "NEW / CREATE"
- "BREAKLOCK"
- "RECREATE"
class TSQLFile : public TFile
class
friend class TBufferSQL2;
friend class TKeySQL;
friend class TSQLStructure;
friend class TSQLTableData;
friend class TSqlRegistry;
friend class TSqlRawBuffer;
friend class TSqlCmdsBuffer;
protected:
enum ELockingKinds { kLockFree = 0, kLockBusy = 1 };
virtual Int_t SysOpen(const char *, Int_t, UInt_t) { return 0; }
virtual Int_t SysClose(Int_t) { return 0; }
virtual Int_t SysRead(Int_t, void *, Int_t) { return 0; }
virtual Int_t SysWrite(Int_t, const void *, Int_t) { return 0; }
virtual Long64_t SysSeek(Int_t, Long64_t, Int_t) { return 0; }
virtual Int_t SysStat(Int_t, Long_t *, Long64_t *, Long_t *, Long_t *) { return 0; }
virtual Int_t SysSync(Int_t) { return 0; }
virtual Long64_t DirCreateEntry(TDirectory *);
virtual Int_t DirReadKeys(TDirectory *);
virtual void DirWriteKeys(TDirectory *);
virtual void DirWriteHeader(TDirectory *);
void SaveToDatabase();
Bool_t ReadConfigurations();
Bool_t IsTablesExists();
void InitSqlDatabase(Bool_t create);
void CreateBasicTables();
void IncrementModifyCounter();
void SetLocking(Int_t mode);
Int_t GetLocking();
Bool_t IsWriteAccess();
Bool_t IsReadAccess();
TSQLResult *SQLQuery(const char *cmd, Int_t flag = 0, Bool_t *res = 0);
Bool_t SQLCanStatement();
TSQLStatement *SQLStatement(const char *cmd, Int_t bufsize = 1000);
void SQLDeleteStatement(TSQLStatement *stmt);
Bool_t SQLApplyCommands(TObjArray *cmds);
Bool_t SQLTestTable(const char *tablename);
Long64_t SQLMaximumValue(const char *tablename, const char *columnname);
void SQLDeleteAllTables();
Bool_t SQLStartTransaction();
Bool_t SQLCommit();
Bool_t SQLRollback();
Int_t SQLMaxIdentifierLength();
void DeleteKeyFromDB(Long64_t keyid);
Bool_t WriteKeyData(TKeySQL *key);
Bool_t UpdateKeyData(TKeySQL *key);
TKeySQL *FindSQLKey(TDirectory *dir, Long64_t keyid);
Long64_t DefineNextKeyId();
Int_t StreamKeysForDirectory(TDirectory *dir, Bool_t doupdate, Long64_t specialkeyid = -1, TKeySQL **specialkey = 0);
TSQLClassInfo *FindSQLClassInfo(const char *clname, Int_t version);
TSQLClassInfo *FindSQLClassInfo(const TClass *cl);
TSQLClassInfo *RequestSQLClassInfo(const char *clname, Int_t version);
TSQLClassInfo *RequestSQLClassInfo(const TClass *cl);
Bool_t CreateClassTable(TSQLClassInfo *sqlinfo, TObjArray *colinfos);
Bool_t CreateRawTable(TSQLClassInfo *sqlinfo);
Bool_t ProduceClassSelectQuery(TVirtualStreamerInfo *info, TSQLClassInfo *sqlinfo, TString &columns, TString &tables,
Int_t &tablecnt);
void AddIdEntry(Long64_t tableid, Int_t subid, Int_t type, const char *name, const char *sqlname, const char *info);
void ReadSQLClassInfos();
TString DefineTableName(const char *clname, Int_t version, Bool_t rawtable);
Bool_t HasTable(const char *name);
TString CodeLongString(Long64_t objid, Int_t strid);
Int_t IsLongStringCode(Long64_t objid, const char *value);
Bool_t VerifyLongStringTable();
Bool_t GetLongString(Long64_t objid, Int_t strid, TString &value);
Long64_t VerifyObjectTable();
Bool_t SQLObjectInfo(Long64_t objid, TString &clname, Version_t &version);
TObjArray *SQLObjectsInfo(Long64_t keyid);
TSQLResult *GetNormalClassData(Long64_t objid, TSQLClassInfo *sqlinfo);
TSQLResult *GetNormalClassDataAll(Long64_t minobjid, Long64_t maxobjid, TSQLClassInfo *sqlinfo);
TSQLResult *GetBlobClassData(Long64_t objid, TSQLClassInfo *sqlinfo);
TSQLStatement *GetBlobClassDataStmt(Long64_t objid, TSQLClassInfo *sqlinfo);
Long64_t StoreObjectInTables(Long64_t keyid, const void *obj, const TClass *cl);
Bool_t WriteSpecialObject(Long64_t keyid, TObject *obj, const char *name, const char *title);
TObject *ReadSpecialObject(Long64_t keyid, TObject *obj = 0);
const char *SQLCompatibleType(Int_t typ) const;
const char *SQLIntType() const;
const char *SQLSmallTextType() const { return fOtherTypes[0]; }
Int_t SQLSmallTextTypeLimit() const { return atoi(fOtherTypes[1]); }
const char *SQLBigTextType() const { return fOtherTypes[2]; }
const char *SQLDatetimeType() const { return fOtherTypes[3]; }
const char *SQLIdentifierQuote() const { return fOtherTypes[4]; }
const char *SQLDirIdColumn() const { return fOtherTypes[5]; }
const char *SQLKeyIdColumn() const { return fOtherTypes[6]; }
const char *SQLObjectIdColumn() const { return fOtherTypes[7]; }
const char *SQLRawIdColumn() const { return fOtherTypes[8]; }
const char *SQLStrIdColumn() const { return fOtherTypes[9]; }
const char *SQLNameSeparator() const { return fOtherTypes[10]; }
const char *SQLValueQuote() const { return fOtherTypes[11]; }
const char *SQLDefaultTableType() const { return fOtherTypes[12]; }
TSQLServer *fSQL;
TList *fSQLClassInfos;
Bool_t fUseSuffixes;
Int_t fSQLIOversion;
Int_t fArrayLimit;
Bool_t fCanChangeConfig;
TString fTablesType;
Int_t fUseTransactions;
Int_t fUseIndexes;
Int_t fModifyCounter;
Int_t fQuerisCounter;
const char **fBasicTypes;
const char **fOtherTypes;
TString fUserName;
std::ofstream *fLogFile;
Bool_t fIdsTableExists;
Int_t fStmtCounter;
private:
void operator=(const TSQLFile &);
public:
enum ETransactionKinds { kTransactionsOff = 0, kTransactionsAuto = 1, kTransactionsUser = 2 };
enum EIndexesKinds { kIndexesNone = 0, kIndexesBasic = 1, kIndexesClass = 2, kIndexesAll = 3 };
TSQLFile();
TSQLFile(const char *dbname, Option_t *option = "read", const char *user = "user", const char *pass = "pass");
virtual ~TSQLFile();
Bool_t GetUseSuffixes() const { return fUseSuffixes; }
void SetUseSuffixes(Bool_t on = kTRUE);
Int_t GetArrayLimit() const { return fArrayLimit; }
void SetArrayLimit(Int_t limit = 20);
void SkipArrayLimit() { SetArrayLimit(-1); }
void SetTablesType(const char *table_type);
const char *GetTablesType() const { return fTablesType.Data(); }
void SetUseTransactions(Int_t mode = kTransactionsAuto);
Int_t GetUseTransactions() const { return fUseTransactions; }
void SetUseIndexes(Int_t use_type = kIndexesBasic);
Int_t GetUseIndexes() const { return fUseIndexes; }
Int_t GetQuerisCounter() const { return fQuerisCounter; }
TString MakeSelectQuery(TClass *cl);
Bool_t StartTransaction();
Bool_t Commit();
Bool_t Rollback();
void StartLogFile(const char *fname);
void StopLogFile();
virtual void Close(Option_t *option = "");
virtual TKey *CreateKey(TDirectory *mother, const TObject *obj, const char *name, Int_t bufsize);
virtual TKey *CreateKey(TDirectory *mother, const void *obj, const TClass *cl, const char *name, Int_t bufsize);
virtual void DrawMap(const char * = "*", Option_t * = "") {}
virtual void FillBuffer(char *&) {}
virtual void Flush() {}
virtual Long64_t GetEND() const { return 0; }
virtual Int_t GetErrno() const { return 0; }
virtual void ResetErrno() const {}
const char *GetDataBaseName() const;
virtual Int_t GetNfree() const { return 0; }
virtual Int_t GetNbytesInfo() const { return 0; }
virtual Int_t GetNbytesFree() const { return 0; }
virtual Long64_t GetSeekFree() const { return 0; }
virtual Long64_t GetSeekInfo() const { return 0; }
virtual Long64_t GetSize() const { return 0; }
virtual TList *GetStreamerInfoList();
Bool_t IsMySQL() const;
virtual Bool_t IsOpen() const;
Bool_t IsOracle() const;
Bool_t IsODBC() const;
virtual void MakeFree(Long64_t, Long64_t) {}
virtual void MakeProject(const char *, const char * = "*", Option_t * = "new") {}
virtual void Map(Option_t *) {}
virtual void Map() {}
virtual void Paint(Option_t * = "") {}
virtual void Print(Option_t * = "") const {}
virtual Bool_t ReadBuffer(char *, Int_t) { return kFALSE; }
virtual Bool_t ReadBuffer(char *, Long64_t, Int_t) { return kFALSE; }
virtual void ReadFree() {}
virtual Int_t Recover() { return 0; }
virtual Int_t ReOpen(Option_t *mode);
virtual void Seek(Long64_t, ERelativeTo = kBeg) {}
virtual void SetEND(Long64_t) {}
virtual Int_t Sizeof() const { return 0; }
virtual Bool_t WriteBuffer(const char *, Int_t) { return kFALSE; }
virtual Int_t Write(const char * = 0, Int_t = 0, Int_t = 0) { return 0; }
virtual Int_t Write(const char * = 0, Int_t = 0, Int_t = 0) const { return 0; }
virtual void WriteFree() {}
virtual void WriteHeader();
virtual void WriteStreamerInfo();
code
auto dbname = "mysql://host.domain:3306/dbname";
auto username = "username";
auto userpass = "userpass";
auto f = new TSQLFile(dbname, "recreate", username, userpass);
arr->Write("arr", TObject::kSingleKey);
h1->Write("histo");
delete f;
auto f = new TSQLFile(dbname, "open", username, userpass);
f->ls();
auto h1 = (TH1*) f->Get("histo");
if (h1!=0) { h1->SetDirectory(0); h1->Draw(); }
auto obj = f->Get("arr");
if (obj!=0) obj->Print("*");
delete f;