SEQ08

SEQ08_processing.ipynb

2018-10-18 Thursday

pause while dDocent is running on all samples

2018-10-17 Wednesday

begin redux
started bcsplit, check back at 2:30pm

If necessary, ssh into the system - ssh michelles@amphiprion.deenr.rutgers.edu

Receive files from sequencer -

use fetch to move them into the /local/shared/pinsky_lab/sequencing/ directory
create a directory for the files, use the naming scheme: type_of_sequencer_year_month_day: hiseq_2015_05_11_SEQ08

Update where files are saved on amphiprion in sample_data sheet

[michelles@amphiprion ~]$ ls /local/shared/pinsky_lab/sequencing/hiseq_2015_05_11_SEQ08/
clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_1_read_1_passed_filter.fastq.gz
clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_1_read_2_index_read_passed_filter.fastq.gz
clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_1_read_3_index_read_passed_filter.fastq.gz
clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_2_read_1_passed_filter.fastq.gz
clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_2_read_2_index_read_passed_filter.fastq.gz
clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_2_read_3_index_read_passed_filter.fastq

Using same directories as SEQ07
make an index file for barcode splitter In the bcsplit directory:

In google sheets, make a file that contains the Pool ID’s in the first column and corresponding illumine index in the second column, separated by tabs.

P030	TGACCA
P031	GCCAAT
P032	CAGATC
P033	CTTGTA

Run barcode splitter on 1st lane from the split directory

[michelles@amphiprion bcsplit]$ nohup barcode_splitter.py --bcfile SEQ08_index.tsv --idxread 2 --suffix .fastq.gz /local/shared/pinsky_lab/sequencing/hiseq_2015_05_11_SEQ08/clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_1_read_1_passed_filter.fastq.gz /local/shared/pinsky_lab/sequencing/hiseq_2015_05_11_SEQ08/clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_1_read_2_index_read_passed_filter.fastq.gz

move all of these files into a "lane 1 directory" just because lane 2 overwrites

1005 cd bcsplit/
1006 ls
1007 mv P03* ./lane1/
1008 ls
1009 mv unmatched-read-1.fastq.gz seq08-unmatched-read-1.fastq.gz
1010 ls
1011 mv unmatched-read-2.fastq.gz seq08-unmatched-read-2.fastq.gz
1012 ls
1013 mv SEQ07_index.tsv ../logs/
1014 ls
1015 less nohup.out
1016 mv nohup.out ../logs/seq08-lane1-bcsplit.out
1017 ls
1018 mv SEQ07-unmatched-read-* ../logs/

Run barcode splitter on 2nd lane from the split directory

[michelles@amphiprion bcsplit]$ nohup barcode_splitter.py --bcfile SEQ08_index.tsv --idxread 2 --suffix .fastq.gz /local/shared/pinsky_lab/sequencing/hiseq_2015_05_11_SEQ08/clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_2_read_1_passed_filter.fastq.gz /local/shared/pinsky_lab/sequencing/hiseq_2015_05_11_SEQ08/clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_2_read_2_index_read_passed_filter.fastq.gz

combine all of the results:

[michelles@amphiprion bcsplit]$ cat ./seq08-unmatched-read-1.fastq.gz ./unmatched-read-1.fastq.gz > SEQ08-unmatched-read1.fastq.gz
[michelles@amphiprion bcsplit]$ cat ./seq08-unmatched-read-2.fastq.gz ./unmatched-read-2.fastq.gz > SEQ08-unmatched-read2.fastq.gz
[michelles@amphiprion bcsplit]$ mv SEQ08* ../logs/
[michelles@amphiprion bcsplit]$ rm seq08-unmatched-read-*
[michelles@amphiprion bcsplit]$ rm unmatched-read-*
[michelles@amphiprion bcsplit]$ rm *-2.fastq.gz
[michelles@amphiprion bcsplit]$ less nohup.out
[michelles@amphiprion bcsplit]$ mv nohup.out seq08-lane2-bcsplit.out
[michelles@amphiprion bcsplit]$ less seq08-lane2-bcsplit.out
[michelles@amphiprion bcsplit]$ mv seq08-lane2-bcsplit.out ../logs/
[michelles@amphiprion bcsplit]$ cd lane1/
[michelles@amphiprion lane1]$ rm *-2.fastq.gz
[michelles@amphiprion bcsplit]$ cat ./lane1/P030-read-1.fastq.gz ./P030-read-1.fastq.gz > P030.fastq.gz
[michelles@amphiprion bcsplit]$ cat ./lane1/P031-read-1.fastq.gz ./P031-read-1.fastq.gz > P031.fastq.gz
[michelles@amphiprion bcsplit]$ cat ./lane1/P032-read-1.fastq.gz ./P032-read-1.fastq.gz > P032.fastq.gz
[michelles@amphiprion bcsplit]$ cat ./lane1/P033-read-1.fastq.gz ./P033-read-1.fastq.gz > P033.fastq.gz

delete extra files

[michelles@amphiprion bcsplit]$ rm -r lane1
[michelles@amphiprion bcsplit]$ rm *1.fastq.gz - deleted P031.fastq.gz - be carefull, use *-1.fastq.gz, not just *fastq.gz!!! -

rerunning bcsplit on seq08 to regenereate Pool 31.

[michelles@amphiprion bcsplit]$ mv ../logs/SEQ08_index.tsv ./
[michelles@amphiprion bcsplit]$ nohup barcode_splitter.py --bcfile SEQ08_index.tsv --idxread 2 --suffix .fastq.gz /local/shared/pinsky_lab/sequencing/hiseq_2015_05_11_SEQ08/clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_1_read_1_passed_filter.fastq.gz /local/shared/pinsky_lab/sequencing/hiseq_2015_05_11_SEQ08/clownfish-ddradseq-seq08-for-231-cycles-h3mgvbcxx_1_read_2_index_read_passed_filter.fastq.gz

Created a barcodes file: from the sample_data spreadsheet, copied the barcodes column from the barcodes tab and pasted into nano, saved as barcodes in 00_working directory
make directories for Pools

[michelles@amphiprion 00_working]$ mkdir 16Pool 17Pool 18Pool 19Pool 30Pool 31Pool 32Pool 33Pool

sign back into the server and run process radtags on the files

978 mv 19process.sh30process.sh
979 nano 30process.sh
980 nohup ./30process.sh
981 cd 00_working/31Pool/
982 cp ../30Pool/30process.sh 31process.sh
983 ls
984 nano 31process.sh
985 nohup ./31process.sh
986 less nohup.out
987 cp 31process.sh ../32Pool/32process.sh
988 cd ../32Pool/
989 ls
990 nano 32process.sh
991 nohup ./32process.sh
992 cd 00_working/
993 ls
994 cd 33
995 cd 33Pool/
996 ls
997 cp ../32Pool/32process.sh 33process.sh
998 ls
999 nano 33process.sh
1000 nohup ./33process.sh

After process_radtags, all of the pools are still in separate directories.

make a new names_barcodes tab delimited file:

Open Sample_data in Chrome (yes, it has to be chrome or firefox, other browsers don’t work properly as of 10-22-2014)
Click on the Pool_contents tab and filter for the pool you are working on (Pool 7)
Copy the sample names over to the first column of the Names tab
drag down the formula to find the barcode
for each pool download as tab delimited "names_Pool7.tsv", etc.
Open the document in Komodo and fix the end of line characters by right clicking on the tab and looking at the properties (doing this with google sheets they were already in Unix)

Copy the names files into the correct pool directories: nrb176:~ Michelle$ scp -r ~/Downloads/names_Pool12.tsv michelles@amphiprion.deenr.rutgers.edu:~/01_seq03/Pool12
Delete intermediate files

[michelles@amphiprion ~]$ cd 01_seq03/bcsplit/lane1
[michelles@amphiprion lane1]$ rm Pool*
[michelles@amphiprion lane1]$ cd ..

Rename files and Move into one directory

[michelles@amphiprion 30Pool]$ sh rename.for.dDocent_se_gz 30Pool_names.tsv
[michelles@amphiprion 30Pool]$ mv APCL1* ../samples/
[michelles@amphiprion 30Pool]$ mv 30* ../logs/
[michelles@amphiprion 32Pool]$ sh rename.for.dDocent_se_gz 32Pool_names.tsv
[michelles@amphiprion 32Pool]$ mv APCL1* ../samples/
[michelles@amphiprion 32Pool]$ mv 32* ../logs/
[michelles@amphiprion 33Pool]$ sh rename.for.dDocent_se_gz 33Pool_names.tsv
[michelles@amphiprion 33Pool]$ mv APCL1* ../samples/
[michelles@amphiprion 33Pool]$ mv 33* ../logs/
michelles@amphiprion 31Pool]$ sh rename.for.dDocent_se_gz 31Pool_names.tsv
[michelles@amphiprion 31Pool]$ mv APCL1* ../samples
[michelles@amphiprion 31Pool]$ mv 31Pool_names.tsv ../logs/

Delete intermediate files:

[michelles@amphiprion 31Pool]$ cd ..
[michelles@amphiprion 00_working]$ rm -r 31Pool/

Run ustacks of seq07 and seq08 together

#!/bin/bash

i=70001
for file in ./samples/*.fq.gz
do
ustacks -t gzfastq -p 20 -d -r -m 3 -M 5 --max_locus_stacks 3 -i $i \
-f ${file} -o ./lax-stacks
let "i+=1";
done

change name of nohup.out

mv nohup.out ./logs/seq07_08_ustacks.out

Do not run cstacks again at this time. Could run cstacks with new files to generate a more comprehensive catalog but we can do that in the future. Currently using the catalog that is made up of the samples with the most loci from SEQ03, SEQ04, SEQ05 generated post rxstacks
copy catalog from the lax-rxstacks folder to the lax-stacks folder

[michelles@amphiprion 00_working]$ cp ./lax-rxstacks/batch_1.catalog.* ./lax-stacks/

Run sstacks (rxstacks needs the matches files). Use the rx-catalogs

[michelles@amphiprion 00_working]$ nohup ./sstacks-lax.sh
#!/bin/bash

for file in $(ls -1 ./lax-stacks/*.tags.tsv.gz \
| grep -v catalog | perl -pe 's/\.tags\.tsv.gz//')
do
    sstacks -p 20 -b 1 -c ./lax-stacks/batch_1 \
            -s $file \
            -o ./lax-stacks/

change name of nohup.out to 05seq_rxstacks.out
Do not run rxcstacks - use the catalog that is made up of the samples with the most loci from SEQ03, SEQ04, SEQ05 generated post rxstacks.
Run sstacks: nohup ./rxsstacks.sh

[michelles@amphiprion 00_working]$ nohup ./populations.sh
#!/bin/bash

#Run sstacks

for file in $(ls -1 ./lax-rxstacks/*.tags.tsv \
| grep -v catalog | perl -pe 's/\.tags\.tsv//')
do
    sstacks -p 25 -b 1 -c ./lax-rxstacks/batch_1 \
            -s $file \
            -o ./lax-rxstacks/
done

change name of nohup.out to 05seq_sstacks.out
Move samples to the lax-stacks and rx-stacks directories in philippines/genotyping for all to use

Run populations on all APCL samples (not just this seq batch)- don’t filter this time around (no -m or -r, etc)

#!/bin/bash

# Calculate population statistics and export several output files.

populations -b 1 -P ./lax-rxstacks/ -s -t 15

change name of nohup.out to YYMMDD_populations.out
Create a mysql database

[michelles@amphiprion ~]$ mysql -plarvae168
mysql> create database seq05;
mysql> show databases;
mysql> exit

Apply stacks configuration to database:

[michelles@amphiprion ~]$ mysql -plarvae168 seq05 < ~/local/share/stacks/sql/stacks.sql

Load radtags to database

[michelles@amphiprion rxstacks]$ load_radtags.pl -D seq05 -p ./ -b 1 -c -B

Index database

[michelles@amphiprion rxstacks]$ index_radtags.pl -D seq05 -c -t

Move the matches, snaps, alleles, and tags files into a combined folder to run all populations together - can this be done with the export script? - Can these files be stored in mysql and then extracted just for this run?
Run populations to produce gene pop file - can’t run all of the samples from multiple runs in the same populations - unless you change the “i” in the script to something that will generate a unique ID???

[michelles@amphiprion 01_seq04]$ cd rxstacks/
[michelles@amphiprion rxstacks]$ nohup populations -b 1 -P ./ -t 10 -r 90 -m 30 -s --genepop where m is the coverage and r is the percent of the population. Start at 90/30, try 90/10, 80/30, 80/10, etc until you get 200 loci

In Windows,
Logged in to windows using the larva password

Started Cervus

Click on Tools>Convert Genotype File>Genepop to Cervus

2 digit format

do not use first ID as population name

Converted 192 individuals in one population at 1132 loci

Clicked on Analysis>Allele Frequency Analysis

Choose the cervus file just created

ID in column 2

First allele in column 3

Number of loci 1132 gives error and says there are 1134 loci, so running that

Save as seq04

OK

Now run a simulation of parentage analysis

Click on analysis>simulation of parentage analysis>sexes of pair unknown

Change parent number to

Saved as seq04simpar

Change to LOD

Ran simulated parentage with 2000 parents, 5000 offspring 0.1 proport samples, .86 typed loci, .01 mismatch, LOD. Computer says it will take 2 hours. - computer froze, dropped it down to 1000 parents, 2000 offspring - took 6+ hours to run
Create parent and offspring files: To make an adult/juvenile file
Run parentage analysis in Cervus - make sure the csv’s have unix line endings in komodo (if only 1 offspring runs, this is the problem).
If running colony, need a marker file that contains the marker IDs, if they are dominant or codominant, and gene dropping, mutation rates.