Installation issues starting with Trinity [SOLVED]

Page 2 of 2 Previous  1, 2

View previous topic View next topic Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by ptranvan on Thu Apr 28, 2016 12:58 pm

Same error:


#####################################################
### Blast 3 : raw reads against unannoted repeats ###
#####################################################
blasting...


Building a new DB, current time: 04/28/2016 12:53:01
New DB name: output_folder2/blast_out/blast3_db.fasta
New DB title: output_folder2/Annotation/unannoted.fasta
Sequence type: Nucleotide
Keep Linkouts: T
Keep MBits: T
Maximum file size: 1000000000B
BLAST options error: File output_folder2/Annotation/unannoted.fasta is empty
Paring blast3 output...
sort: échec d'ouverture: output_folder2/blast_out/reads_vs_unannoted.blast.out: Aucun fichier ou dossier de ce type
#######################################################
### Estimation of Repeat content from blast outputs ###
#######################################################
parsing blastout and adding RM annotations for each read...
join: contigsTrinityRM.sorted: Aucun fichier ou dossier de ce type
Done, results in: blast_out/blastout_final_fmtd_annoted

I've downloaded the new version, and executed your command in another output ...

ptranvan

Posts : 13
Join date : 2016-04-22

View user profile

Back to top Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by Clément Goubert on Thu Apr 28, 2016 1:07 pm

Hello,

Can you send me the whole log and the exact command line you ran? Also, can you tell me which kind of computing facility your'e using?
Thanks!

Clément
avatar
Clément Goubert
Admin

Posts : 30
Join date : 2016-01-05
Age : 29

View user profile https://lbbe.univ-lyon1.fr/-dnaPipeTE-.html

Back to top Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by ptranvan on Thu Apr 28, 2016 1:42 pm

My command:

Code:

python3 dnaPipeTE.py -input test_dataset.fastq -output output_folder2 -genome_size 10000000 -genome_coverage 0.1 -sample_number 1

My log:


Let's go !!!

Start time: Thu Apr 28 11:37:42 2016
total number of reads: 100125
maximum number of reads to sample: 12048
fastq : test_dataset.fastq
sampling 1 samples of max 12048 reads to reach coverage...
999984 bases sampled in 12048 reads
s_test_dataset.fastq done.
###################################
### TRINITY to assemble repeats ###
###################################

***** TRINITY iteration 1 *****

Selecting reads for Trinity iteration number 1...
awk: cmd. ligne:1: Fatal: ne peut ouvrir le fichier « output_folder2/Trinity_run0/chrysalis/readsToComponents.out.sort » en lecture (Aucun fichier ou dossier de ce type)
Done

Current settings:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 8271084
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 12288
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


-since butterfly will eventually be run, lets test for proper execution of java
#######################################
Running Java Tests
Thursday, April 28, 2016: 11:37:44 CMD: java -Xmx64m -jar /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/util/support_scripts/ExitTester.jar 0
CMD finished (1 seconds)
Thursday, April 28, 2016: 11:37:45 CMD: java -Xmx64m -jar /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/util/support_scripts/ExitTester.jar 1
-we properly captured the java failure status, as needed. Looking good.
Java tests succeeded.
###################################

Thursday, April 28, 2016: 11:37:45 CMD: mkdir -p /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis
CMD finished (0 seconds)
Thursday, April 28, 2016: 11:37:45 CMD: ln -s /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/s0_test_dataset.fastq.fasta single.fa
CMD finished (0 seconds)
-------------------------------------------
----------- Jellyfish --------------------
-- (building a k-mer catalog from reads) --
-------------------------------------------

Thursday, April 28, 2016: 11:37:45 CMD: /software/UHTS/Analysis/jellyfish/1.1.11//bin/jellyfish count -t 1 -m 25 -s 1533763575 --both-strands single.fa
CMD finished (42 seconds)
Thursday, April 28, 2016: 11:38:27 CMD: /software/UHTS/Analysis/jellyfish/1.1.11//bin/jellyfish dump -L 1 mer_counts_0 > jellyfish.kmers.fa
CMD finished (1 seconds)
Thursday, April 28, 2016: 11:38:28 CMD: /software/UHTS/Analysis/jellyfish/1.1.11//bin/jellyfish histo -t 1 -o jellyfish.kmers.fa.histo mer_counts_0
CMD finished (0 seconds)
Thursday, April 28, 2016: 11:38:28 CMD: touch jellyfish.1.finished
CMD finished (0 seconds)
----------------------------------------------
--------------- Inchworm ---------------------
-- (Linear contig construction from k-mers) --
----------------------------------------------

Thursday, April 28, 2016: 11:38:28 CMD: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/Inchworm/bin/inchworm --kmers jellyfish.kmers.fa --run_inchworm -K 25 -L 25 --monitor 1 --DS --keep_tmp_files --num_threads 1 --PARALLEL_IWORM > /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/inchworm.K25.L25.DS.fa.tmp
Kmer length set to: 25
Min assembly length set to: 25
Monitor turned on, set to: 1
-retaining tmp files
double stranded mode set
setting number of threads to: 1
-setting parallel iworm mode.
-reading Kmer occurences...
[0M] Kmers parsed.
done parsing 631283 Kmers, 631283 added, taking 2 seconds.

TIMING KMER_DB_BUILDING 2 s.
Pruning kmers (min_kmer_count=1 min_any_entropy=0 min_ratio_non_error=0.05)
Pruned 131 kmers from catalog.
Pruning time: 0 seconds = 0 minutes.

TIMING PRUNING 0 s.
-populating the kmer seed candidate list.
Kcounter hash size: 631283
Processed 631152 non-zero abundance kmers in kcounter.
WARNING: Not sorting list of kmers. This is OK if PARALLEL_IWORM is in effect.
-beginning inchworm contig assembly.
Total kcounter hash size: 631283 vs. sorted list size: 631152
num threads set to: 1
Done opening file. tmp.iworm.fa.pid_17121.thread_0

Iworm contig assembly time: 3 seconds = 0.05 minutes.

TIMING CONTIG_BUILDING 3 s.

TIMING PROG_RUNTIME 5 s.
CMD finished (5 seconds)
Thursday, April 28, 2016: 11:38:33 CMD: touch /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/inchworm.K25.L25.DS.fa.finished
CMD finished (0 seconds)
Thursday, April 28, 2016: 11:38:33 CMD: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/Chrysalis/Chrysalis -i single.fa -iworm /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/inchworm.K25.L25.DS.fa -o /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis -cpu 1 -min_glue 1 -min_iso_ratio 0.05 -glue_factor 0.05 -weldmer_size 48 -min 200 -dist 500 -max_reads 200000 -sort_buffer_size 10G -max_mem_reads 10000000 -butterfly /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/Butterfly/Butterfly.jar 2>&1
Path to executables: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/Chrysalis/
-------------------
---- Chrysalis ----
-------------------

-- running Chryalis: GraphFromFasta --
Running: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/Chrysalis/GraphFromFasta -i /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/inchworm.K25.L25.DS.fa -r single.fa -min_contig_length 200 -min_glue 1 -glue_factor 0.05 -min_iso_ratio 0.05 -t 1 -kk 48 > /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/GraphFromIwormFasta.out
-----------------------------------------
--- Chrysalis: GraphFromFasta -----------
-- (cluster related inchworm contigs) ---
-----------------------------------------

-setting num threads to: 1
-running on 1 threads
GraphFromFasta: Reading file: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/inchworm.K25.L25.DS.fa
done!
Counting k-mers...
KmerAlignCore- Contigs: 244
done, assigning k-mers...
KmerAlignCore- Contigs: 244
done!
-setting omp for schedule chunksize to 2 for 245 iworm contigs
Phase 1: Collecting candidate weldmers between iworm contig pairs sharing k-1mers
Processed: 99.5918 % of iworm contigs.

...done Phase 1. (0.0810258 seconds)
Setting up reads for streaming...
Identifying reads that support welding of iworm contigs...
Reads: (unknown count: streaming-mode)


Done!
Phase 2: Reclustering iworm contigs using welds.
[82.0408% done]
now bubbling:

done bubbling.
-- writing inchworm bundled.fasta and computing & partitioning component graph files for parallel processing.
-- mapping reads to chrysalis components.
Running: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/Chrysalis/ReadsToTranscripts -i single.fa -f /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/bundled_iworm_contigs.fasta -t 1 -o /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis -max_mem_reads 10000000
-------------------------------------------
---- Chrysalis: ReadsToTranscripts --------
-- (Place reads on Inchworm Bundles) ------
-------------------------------------------

Setting maximum number of reads to load in memory to 10000000
-setting num threads to: 1
Reading bundled inchworm contigs...
done!
Assigning kmers to Iworm bundles ... done!
Processing reads:
reading another 10000000... done. Read 12048 reads.
[12048] reads analyzed for mapping.
[8] components written.
reading another 10000000... finished reading reads
Done
31660 bytes
CMD: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/Chrysalis//../trinity-plugins/coreutils/bin/sort --parallel=1 -T . -S 10G -k 1,1n /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/readsToComponents.out > /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/readsToComponents.out.sort
31660 bytes
CMD finished (1 seconds)
Chrysalis initial stage completed successfully.
Thursday, April 28, 2016: 11:38:34 CMD: touch /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/chrysalis.finished
CMD finished (0 seconds)
Thursday, April 28, 2016: 11:38:34 CMD: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/util/support_scripts/partition_chrysalis_graphs_n_reads.pl --deBruijns /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/bundled_iworm_contigs.fasta.deBruijn --componentReads /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/readsToComponents.out.sort -N 1000 -L 200 --compdir /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/Component_bins
Partitioning chrysalis graphs and reads


Done partitioning graphs.
Partitioning reads...

Done partitioning reads.

-writing /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/component_base_listing.txt

Done.

CMD finished (1 seconds)
Thursday, April 28, 2016: 11:38:35 CMD: touch /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/file_partitioning.ok
CMD finished (0 seconds)
---------------------------------------------------
----------- Chrysalis: QuantifyGraph --------------
-- (Integrate mapped reads into de Bruijn graph) --
---------------------------------------------------

Thursday, April 28, 2016: 11:38:35 CMD: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/trinity-plugins/parafly/bin/ParaFly -c /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/quantifyGraph_commands -CPU 1 -failed_cmds failed_quantify_graph_commands.16602.txt -v -shuffle
Number of Commands: 8
succeeded(Cool 100% completed.

All commands completed successfully. :-)

CMD finished (0 seconds)
Thursday, April 28, 2016: 11:38:35 CMD: touch /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/quantifyGraph_commands.run.finished
CMD finished (0 seconds)
Butterfly_cmds: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/butterfly_commands
Inchworm and Chrysalis complete. Butterfly commands to execute are provided here:
/stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/butterfly_commands

---------------------------------------------------------------
-------------------- Butterfly --------------------------------
-- (Reconstruct transcripts from reads and de Bruijn graphs) --
---------------------------------------------------------------

Thursday, April 28, 2016: 11:38:35 CMD: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/trinity-plugins/parafly/bin/ParaFly -c /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/butterfly_commands -shuffle -CPU 1 -failed_cmds failed_butterfly_commands.16602.txt -v
Number of Commands: 8
succeeded(Cool 100% completed.

All commands completed successfully. :-)

CMD finished (5 seconds)
Thursday, April 28, 2016: 11:38:40 CMD: /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/bin/trinityrnaseq_r20140413p1/util/support_scripts/print_butterfly_assemblies.pl /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/chrysalis/component_base_listing.txt > Trinity.fasta.tmp
CMD finished (0 seconds)


###################################################################
Butterfly assemblies are written to /stn4/ul/monthly/ptranvan/Software/dnaPipeTE-master/output_folder2/Trinity_run1/Trinity.fasta
###################################################################


Trinity iteration 1 Done'
renaming Trinity output...
done
output_folder2/Annotation/one_RM_hit_per_Trinity_contigs
output_folder2/Annotation/Best_RM_annot_80-80
output_folder2/Annotation/Best_RM_annot_partial
#######################################
### REPEATMASKER to anotate contigs ###
#######################################

RepeatMasker version open-4.0.6
Search Engine: NCBI/RMBLAST [ 2.2.27+ ]
Master RepeatMasker Database: ./bin/RepeatMasker/Libraries/RepeatMaskerLib.embl ( Complete Database: 20150807 )



analyzing file output_folder2/Trinity.fasta

Checking for E. coli insertion elements
identifying Simple Repeats in batch 1 of 1
identifying matches to root sequences in batch 1 of 1
identifying Simple Repeats in batch 1 of 1
processing output:
cycle 1
cycle 2
cycle 3
cycle 4
cycle 5
cycle 6
cycle 7
cycle 8
cycle 9
cycle 10
Generating output...
masking
done
19 line read, sorting...
sort done, filtering...
10 lines in one_RM_hit_per_Trinity_contigs
0 lines in Best_RM_annot_80
6 lines in Best_RM_annot_partial
Done
#########################################
### Making contigs annotation from RM ###
#########################################
Done


Making blast sample...
sampling file found, skipping sampling...
total number of reads: 100125
maximum number of reads to sample: 12048
fastq : test_dataset.fastq
sampling 1 samples of max 12048 reads to reach coverage...
999984 bases sampled in 12048 reads
s_test_dataset.fastq_blast done.
#######################################################
### Blast 1 : raw reads against all repeats contigs ###
#######################################################
blasting...


Building a new DB, current time: 04/28/2016 13:39:04
New DB name: output_folder2/Trinity.fasta
New DB title: output_folder2/Trinity.fasta
Sequence type: Nucleotide
Keep Linkouts: T
Keep MBits: T
Maximum file size: 1000000000B
Adding sequences from FASTA; added 10 sequences in 0.00227189 seconds.
When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; and it won't cost you a cent.

To silence this citation notice run 'parallel --bibtex' once or use '--no-notice'.

Paring blast1 output...
###################################################
### Blast 2 : raw reads against annoted repeats ###
###################################################
blasting...


Building a new DB, current time: 04/28/2016 13:39:18
New DB name: output_folder2/blast_out/blast2_db.fasta
New DB title: output_folder2/blast_out/blast2_db.fasta
Sequence type: Nucleotide
Keep Linkouts: T
Keep MBits: T
Maximum file size: 1000000000B
Adding sequences from FASTA; added 35744 sequences in 39.835 seconds.
When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; and it won't cost you a cent.

To silence this citation notice run 'parallel --bibtex' once or use '--no-notice'.

Paring blast2 output...
Selecting non-matching reads for blast3
#####################################################
### Blast 3 : raw reads against unannoted repeats ###
#####################################################
blasting...


Building a new DB, current time: 04/28/2016 13:40:31
New DB name: output_folder2/blast_out/blast3_db.fasta
New DB title: output_folder2/Annotation/unannoted.fasta
Sequence type: Nucleotide
Keep Linkouts: T
Keep MBits: T
Maximum file size: 1000000000B
BLAST options error: File output_folder2/Annotation/unannoted.fasta is empty
Paring blast3 output...
sort: échec d'ouverture: output_folder2/blast_out/reads_vs_unannoted.blast.out: Aucun fichier ou dossier de ce type
#######################################################
### Estimation of Repeat content from blast outputs ###
#######################################################
parsing blastout and adding RM annotations for each read...
join: contigsTrinityRM.sorted: Aucun fichier ou dossier de ce type
Done, results in: blast_out/blastout_final_fmtd_annoted
#########################################
### OK, lets build some pretty graphs ###
#########################################
Drawing graphs...
null device
1
null device
1
null device
1
null device
1
Warning message:
Removed 4 rows containing missing values (geom_bar).
Warning message:
Removed 4 rows containing missing values (geom_bar).
Done
Removing Trinity runs files...
done
Finishin time: Thu Apr 28 11:40:33 2016
########################
# see you soon !!! #
########################

ptranvan

Posts : 13
Join date : 2016-04-22

View user profile

Back to top Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by Clément Goubert on Thu Apr 28, 2016 2:02 pm

Ok, actually I also re-tried here with the same parameters, and I have the same results.
This is normal, there is no read that match "NA" contigs, because of the small sample size. Since the test dataset is Drosophilas, almost all repeats are annotated and so with a small sample there is no hit at blast 3.

For the rest everything seems fine. You can check that your results are consistent with the one of the test folder, where an example output is provided !

Cheers,

Clément
avatar
Clément Goubert
Admin

Posts : 30
Join date : 2016-01-05
Age : 29

View user profile https://lbbe.univ-lyon1.fr/-dnaPipeTE-.html

Back to top Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by ptranvan on Fri Apr 29, 2016 1:13 pm

OK, so now I've tried with my sample:

Code:

python3 ../dnaPipeTE.py -input myseq.fq -output output


Start time: Fri Apr 29 11:12:27 2016
total number of reads: 273303
maximum number of reads to sample: 540000
fastq : myseq.fq
Traceback (most recent call last):
File "../dnaPipeTE.py", line 697, in <module>
Sampler = FastqSamplerToFasta(args.input_file, args.sample_size, args.genome_size, args.genome_coverage, args.sample_number, args.output_folder, False)
File "../dnaPipeTE.py", line 156, in __init__
self.get_sampled_id(self.fastq_R1)
File "../dnaPipeTE.py", line 220, in get_sampled_id
tirages = random.sample(range(np), self.number*self.sample_number)
File "/software/lib64/python3.4/random.py", line 315, in sample
raise ValueError("Sample larger than population")
ValueError: Sample larger than population

How to fix that

ptranvan

Posts : 13
Join date : 2016-04-22

View user profile

Back to top Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by Clément Goubert on Fri Apr 29, 2016 3:30 pm

Hi,

Here it seems that you don't have enough reads in your dataset to reach the desired coverage.
Are you using v1.2? We fixed a bug that is in 1.1 about sampling: even if you actually have the right number of bases in your dataset, it used to break if according to the size of the smallest reads, you won't have enough of the smallest to reach the desired coverage, but now, that is fixed in 1.2

Clément
avatar
Clément Goubert
Admin

Posts : 30
Join date : 2016-01-05
Age : 29

View user profile https://lbbe.univ-lyon1.fr/-dnaPipeTE-.html

Back to top Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by ptranvan on Fri Apr 29, 2016 4:08 pm

Yes, I use the last version ...
What is the minimun number of reads required to run your program then ?

ptranvan

Posts : 13
Join date : 2016-04-22

View user profile

Back to top Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by Clément Goubert on Mon May 02, 2016 11:00 am

Well, it really depends your model. For a Drosophila genome, that is approx 120 Mpb, I use a sample coverage of 0.25X.
Using 0.1X in larger genome usually works fine but you'll miss the less repeated elements.
avatar
Clément Goubert
Admin

Posts : 30
Join date : 2016-01-05
Age : 29

View user profile https://lbbe.univ-lyon1.fr/-dnaPipeTE-.html

Back to top Go down

Re: Installation issues starting with Trinity [SOLVED]

Post by Sponsored content


Sponsored content


Back to top Go down

Page 2 of 2 Previous  1, 2

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum