Install VariantDB on local hardware

Prerequisites

  1. Operating system: Installation was tested on Ubuntu 12.04 LTS, 64bit.
  2. LAMP server with Perl: MySQL-databases can be located on remote servers.
  3. Mercurial : Used for installing the system
  4. Memcache (optional)
  5. ProFTPd (optional)

Install the VariantDB basics

  1. Clone the Installation Scripts: `hg clone https://bitbucket.org/medgenua/vdb_installer`
  2. Enter the cloned directory : `cd vdb_installer`
  3. Install databases: `sudo ./Install_databases.sh`
    1. Provide a mysql username and password to access VariantDB data. By default, a new user with random password is generated.
    2. Provide the mysql host that will host the VariantDB data
    3. Wild card access can be allowed to the VariantDB databases by using %
    4. Enter your mysql root password to generate the user.
    5. The program will now install a base version of the database
    6. DB-credentials are written to a file in the current folder: .dbpass.txt. Note them down for the next step.
  4. Install the web-interface: `sudo ./Install_Web_Interface.sh`
    1. Install dependencies : tofrodos is done automatically, others you have to agree for
    2. Install cpan Modules: done automatically. They are only installed if missing, not updated. Make sure to use the 'sudo' approach of cpan instead of local::lib if initial configuration is run. Otherwise, the modules might not be available to the dedicated runtime user!
    3. Provide Installation options (all documented in installer output)
    4. The system is downloaded (by mercurial) and installed
    5. The annotation databases are downloaded (from ucsc/annovar/snpEff). This totals in approximately 500Gb ! The first run installs a core set of annotations. Afterwards you get information on how to enable old and very large annotation sests, or how to install them anyway.
    6. Provide an email to perform administrative tasks.
    7. Provide a user to execute the annotation tasks. All VariantDB data will be owned by this user. By default, a new user with random password is generated.
    8. Provide the mysql user created in the previous step (install_databases) if asked. If you install databases and web from the same machine/folder, credentials are read in automatically. The credentials are available in the file .dbpass.txt
    9. An edit to sudoers.d is made to allow www-user to switch to the script user
    10. cgi-bin configuration is updated
    11. php.ini is updated
    12. You're done.

Tune Database

  1. VariantDB uses MyISAM databases. The following variables optimize the mysql server for large MyISAM tables.
  2. Open mySQL config file (/etc/mysql/my.cnf) on the databases server
  3. Edit the following settings to settings appriate for your server:
    1. [mysqld]
    2. key_buffer_size = 4048M
    3. max_allowed_packet = 512M
    4. thread_stack = 256K
    5. thread_cache_size = 8
    6. tmp_table_size = 2048M
    7. max_heap_table_size = 2048M
    8. bulk_insert_buffer_size = 128M
    9. connect_timeout = 15
    10. local_infile <= MANDATORY
    11. myisam_repair_threads = 4
    12. myisam_sort_buffer_size = 4048M
    13. open_files_limit = 8000
    14. query_cache_limit = 12M
    15. query_cache_size = 128M
    16. read_buffer_size = 16M
    17. read_rnd_buffer_size = 8M
    18. open_files_limit = 8000
  4. Restart MySQL server

Start Annotation Monitor

  1. The following script monitors triggers for new data and launches annotation jobs.
  3. open /etc/rc.local for editing (sudo vim /etc/rc.local)
  4. Just before 'exit 0', add the following line (%scriptuser% is listed in VariantDB/.Credentials/.credentials file):
      su %scriptuser% -c 'cd /path_to/VariantDB/Web-Interface/Annotations/ && perl monitor.pl > /dev/null 2>&1 &'
  5. Reboot the system

Initialize VariantDB

  1. Go to your installation : http://<domain>/<path>
  2. Create the first user. This user will be a platform administrator
  3. Once you submit this form, access to VariantDB is activated for all users.

Set up ProFTPD

Note: Guidelines apply to Ubuntu 14.04 LTS

  1. Install Proftp (version ≥ 1.3.4) and mysql module : apt-get install proftpd proftpd-mod-mysql
  2. Activate mysql modules in /etc/proftpd/proftpd.conf:
    LoadModule mod_sql.c
LoadModule mod_sql_mysql.c
LoadModule mod_sql_passwd.c
  3. Add to the /etc/proftpd/proftpd.conf:
    # chroot users to their homes
DefaultRoot ~
# Create home directories as needed, using chmod 755
# chmod 755 is set to make them www-user readable. Adapt to use groups
CreateHome on 755 dirmode 755
#activate mysql backend
SQLEngine on
SQLBackend mysql
# Set up mod_sql_password - VariantDB passwords are stored as sha1-encoded 
SQLPasswordEngine  on
SQLPasswordEncoding hex 
AuthOrder mod_sql.c
# adapt 'NGS-Variants".$_SESSION['dbname']."' after a Genome Build update
SQLConnectInfo NGS-Variants-hg19@<DB-HOST> <DBUSER> <DBPASS>
SQLAuthTypes SHA1
SQLAuthenticate users
# set default homedir to some existing folder. It's not actually used.
SQLDefaultHomedir /tmp
# define the query to authenticate users against VariantDB database.
SQLUserInfo custom:/LookupVariantDBUser
SQLNamedQuery LookupVariantDBUser SELECT "email, password_sha1, '<scriptuser_UID>', '<scriptuser_GID>', '<path_to_variant_db_ftp-data>/%U', '/bin/bash' FROM Users WHERE email = '%U'"
  4. The parameters to replace are found in /<VariantDB>/.Credentials/.credentials file
  5. Restart proftpd : sudo /etc/init.d/proftpd restart
  6. Set the following lines in /VariantDB/.Credentials/.credentials
    FTP=1
FTP_HOST=YOUR FTP SERVER IP/NAME
FTP_PORT=YOUR FTP SERVER PORT
FTP_DATA=/path/to/the/ftp/data/base/dir