Home » Download files from website with many links [closed]

Download files from website with many links [closed]

Solutons:


If all the links are as regular as they look, you can try this:

results=147
for ((i=0; i<147; i++)); do
    base=ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR445/
    ext=$((445410+i))
    suff=.fastq.gz
    for j in 1 2; do wget "${base}${ext}/${ext}/_${j}${suff}"; done
done

There is nothing recursive in this download as far as I can see. Link address inspection gives some hint:

ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR445/ERR445410/ERR445410_1.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR445/ERR445410/ERR445410_2.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR445/ERR445411/ERR445411_1.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR445/ERR445411/ERR445411_2.fastq.gz

So there should be only three parts that change between them:

ERR<nr>/ERR<nr>_<1 or 2>

The stepping for ERR is 1 and the result count is 147, which makes for a complete answer, assuming that’s all in this story. So after you’ve finished all downloads, count new files and make sure you have 2 * 147 of them.

The download procedure can certainly be optimised. As it is above, there’s one wget process started for each link. I’ll leave the optimisation up to you, presenting here just the analysis and the basic solution.

Related Solutions

Why not drop the “auto” keyword? [duplicate]

Your proposal would be rejected on the basis of backward compatibility alone. But let's say for the sake of argument that the standards committee like your idea. You don't take into account the numerous ways you can initialize a variable widget w; // (a) widget...

Recursive to iterative using a systematic method [closed]

So, to restate the question. We have a function f, in our case fac. def fac(n): if n==0: return 1 else: return n*fac(n-1) It is implemented recursively. We want to implement a function facOpt that does the same thing but iteratively. fac is written almost in...

How can I match values in one file to ranges from another?

if the data file sizes are not huge, there is a simpler way $ join input1 input2 | awk '$5<$4 && $3<$5 {print $2, $5-$3+1}' B100002 32 B100043 15 B123465 3 This Perl code seems to solve your problem It is a common idiom: to load the entire...

Javascript difference between “=” and “===” [duplicate]

You need to use == or === for equality checking. = is the assignment operator. You can read about assignment operators here on MDN. As a quick reference as you are learning JS: = assignment operator == equal to === equal value and equal type != not equal !==...

Compiler complains about misplaced else [closed]

Your compiler complains about an misplaced else because, well, there is an else without a preceding if: // ... for (j=1; j<n-i; j++) { if(a[j]<=a[j+1]) { // ... } // END OF IF } // END OF FOR else { continue; } // ... The else in your code does not follow...

Bootstrap – custom alerts with progress bar

/* !important are just used to overide the bootstrap css in the snippet */ .alertContainer { border-radius: 0 !important; border-width: 0 !important; padding: 0 !important; height: auto !important; position: absolute !important; bottom: 15px !important; left:...

How to Garbage Collect an external Javascript load?

Yes, s.onload = null is useful and will garbage collect! As of 2019, it is not possible to explicitly or programmatically trigger garbage collection in JavaScript. That means it collects when it wants. Although there is cases where setting to null may do a GC...

Math programming with python

At first, what you are looking for is the modulo operator and the function math.floor() Modulo from wikipedia: In computing, the modulo operation finds the remainder after division of one number by another (sometimes called modulus). for example: 12%12=0...

Android slide over letters to create a word [closed]

Here some advice you can use: First for each cell you can create an object that represents the state of that cell: class Cell { char mChar; int row,column; boolean isSelected; } then you can create a 2D array of your cells Cell[][] mTable = ... For views you...

Sum two integers in Java

You reused the x and y variable names (hence the variable x is already defined in method main error), and forgot to assign the ints read from the Scanner to the x and y variables. Besides, there's no need to create two Scanner objects. public static void...

Extend three classes that implements an interface in Java

Using this simplified implementation of the library, using method() instead of M(): interface IFC { void method(); } class A implements IFC { public void method() { System.out.println("method in A"); }; } As akuzminykh mentions in their comment You'd write a...

How to set the stream content in PHPExcel? [closed]

Okey, First thing first PHPExcel_Worksheet_MemoryDrawing() can't solve your problem if you insist to use stream content and pass that to your worksheet your PDF will not render your image. But you can use `PHPExcel_Worksheet_Drawing()' if you want to render...

How to remove all files from a directory?

Linux does not use extensions. It is up to the creator of the file to decide whether the name should have an extension. Linux looks at the first few bytes to figure out what kind of file it is dealing with. To remove all non-hidden files* in a directory use: rm...

Hacker used picture upload to get PHP code into my site

Client side validation The validation code you have provided is in JavaScript. That suggests it is code that you use to do the validation on the client. Rule number one of securing webapps is to never trust the client. The client is under the full control of...

First Time HTML5/CSS Site

Semantically, I would suggest using HTML5 elements more. For example, instead of... <div id="header"> <div id="logo"></div> </div> Use instead: (the ID can stay if you want it to) <header> <div id="logo"></div>...