Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length

 
Advanced search

879123 Posts in 32961 Topics- by 24354 Members - Latest Member: saga design

May 23, 2013, 10:31:32 AM
TIGSource ForumsCommunityCompetitionsOld CompetitionsAssemblee: Part 1Assembleech - The Assemblee Link Leech (19/12 update)
Pages: 1 2 [3] 4 5
Print
Author Topic: Assembleech - The Assemblee Link Leech (19/12 update)  (Read 16633 times)
Fifth
Level 10
*****


yoster_5@hotmail.com
View Profile Email
« Reply #30 on: November 27, 2009, 07:44:19 PM »

You could try the threads in print mode.  You'd get all of the posts in one page with the author's name up at top.
Logged
nitram_cero (2bam)
Level 10
*****


DIY


View Profile WWW Email
« Reply #31 on: November 28, 2009, 12:42:16 PM »

Well, if someone manages to do before tomorrow it it would be cool.

Here's the source of the download tool and the correct wget for Windows (command-line, not gnuwin32's GUI one).

Src + Wget.exe

GET THE ONE FROM THE FIRST POST! Smiley

You can compile it in Mac easily with:
Quote
$ g++ -Wall leech_tool.cpp -o leech_tool

And in Windows w/MSVC creating a "Console app/Empty Project" solution,
dragging the leech_tool.cpp to the "Sources" tree folder and compiling.

I could get you executables but I need to know the leech link to directly compile it there (to avoid people having to open a console/terminal and writting the link themselves)

Or here's the code if you're too lazy to download the zip

Code:
// leech_tool:
// Tool to download an "<author>\n<url>\n<author>\n<url>\n"... file list and then
// put each file under the correct author folder.
//
// It was built to work with Indie Asset Organizer and
// download everything using the Assembleech script (by "Sos", Thanks to you!)
//
// Released under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 license.
//
// You need to have wget on the same folder where this is run from.

//v1.0

#include <stdlib.h>
#include <iostream>
#include <fstream>
#include <string>
#include <list>
#include <sstream>
#include <ctype.h>
using namespace std;

#error You must define a URL before you compile
#define URL_LEECH //"http://so0os.pl/assembleech/"
#define TEMP_LIST "./temp-list.txt"


//composite string, ret code
list<pair<string, int>> failed;

const char * const wgetExitCodes[] = {
    "No problems occurred.",
    "Generic error code.",
    "Parse error—for instance, when parsing command-line options, the ‘.wgetrc’ or ‘.netrc’...",
    "File I/O error.",
    "Network failure.",
    "SSL verification failure.",
    "Username/password authentication failure.",
    "Protocol errors.",
    "Server issued an error response.",
};

void getFile(const string &author, const string &url)
{
string::size_type slash=url.find_last_of('/');
string filename = slash != string::npos ? url.substr(slash+1) : url;

cout << author << "'s " << filename << "...";

//quiet, continue, no host directories and output to author directory
stringstream call;
call << "wget -q -c -nH -P ./assets/" << author << "/ " << url;

int ret;
if((ret=system(call.str().c_str())) != 0) {
stringstream failStr;
failStr << author << "'s " << url;
failed.push_back(make_pair(failStr.str(), ret));
cout << "FAIL" << endl;
}
else
cout << "OK" << endl;
}

bool getList()
{
cout << "Retrieving full list..." << endl;
int ret;
//quiet and output to specific path
if((ret=system("wget -q -O " TEMP_LIST " " URL_LEECH)) != 0) {
cout << "Couldn't retrieve list. wget error "<<ret<<(0<=ret && ret<=8 ? wgetExitCodes[ret] : "???")<<endl;
return false;
}

ifstream flist(TEMP_LIST);
string author;
string url;
while(!flist.eof()) {
getline(flist, author);
getline(flist, url);

//Fix to work with IAO
for(string::size_type i=0; i<author.length(); i++) {
if(isspace(author[i]))
author[i] = '_';
}

getFile(author, url);
}

return true;
}



int main()
{
cout << "leech_tool: download everything from the assembleech by Sos to" << endl;
cout << " Indie Asset Organizer compatible folders" << endl;
cout << endl;
cout << "Links must be direct for this to work" << endl;
cout << "Failed links will be listed last" << endl;
cout << endl;

if(!getList())
return 1;

cout << "================== FAILED LIST ==================" << endl;
for(list<pair<string, int>>::iterator it = failed.begin(); it != failed.end(); ++it) {
cout << "FAILED(" << " - " << it->second
<< ": " << (0<=it->second && it->second<=8 ? wgetExitCodes[it->second] : "???")
<< "): " << it->first << endl;
}

cout << endl << "Press enter to exit" << endl;
getchar(); //Mac friendly I believe.
return 0;
}
« Last Edit: December 02, 2009, 03:13:11 AM by nitram_cero » Logged

2BAM.com indie games
Craig Stern
Level 10
*****


I'm not actually all that stern.


View Profile WWW Email
« Reply #32 on: November 28, 2009, 11:35:30 PM »

Dude, this is awesome! The Assembleech Search seems to have problems when I search for mp3, though. Its results for this thread, for example, are

Quote

Just so you know.
Logged

Tesse
Level 0
**



View Profile
« Reply #33 on: November 29, 2009, 03:31:27 AM »

I don't know if my idea is a good one or not but...

I've downloaded all the links with the DownThemAll plugin for Firefox.
Now I have a folder with more than 1200 files and I don't know where do they come etc.

I don't know how to code PHP, otherwise I would already have done it.
I think basically that every link pointing to a png/jpg/gif should be displayed as an image that links to the orginal post when clicked on it.

Yes, would takes time to load but if it's to have a complete gallery of what you are looking for, maybe it worth it.
Logged

Please be friend of me.
nitram_cero (2bam)
Level 10
*****


DIY


View Profile WWW Email
« Reply #34 on: November 29, 2009, 01:10:26 PM »

I made the Indie Asset Organizer almost exclusively for this competition.
It went through almost unnoticed but it has really useful features. It even runs on Mac.

If there was a way to work out the assembleech to output authors and file links, leech_tool downloads everything into author-named subfolders and can be used by the Indie Asset Org. to be easily previewed (audio and graphics), drag-copied, etc.

It's a shame I spent time doing those tools and nobody is using them. Oh well.



« Last Edit: November 29, 2009, 01:14:02 PM by nitram_cero » Logged

2BAM.com indie games
Sos
Level 8
***


[cluck]


View Profile WWW Email
« Reply #35 on: November 29, 2009, 02:01:22 PM »

Sorry for not having that done already. I was a bit busy, and now I'm away. will do it tmrw.
Logged

Pencerkoff
CCCP
Level 4
*


Hello I am Pencerkoff


View Profile
« Reply #36 on: November 29, 2009, 07:53:20 PM »

Hello this is Pencerkoff

It's a shame I spent time doing those tools and nobody is using them. Oh well.

Yeah, too bad you spent time expanding your horizons by creating something instead of playing Call of Duty 4.

-PENCERKOFF
Logged

nitram_cero (2bam)
Level 10
*****


DIY


View Profile WWW Email
« Reply #37 on: November 29, 2009, 08:31:37 PM »

Yeah, too bad you spent time expanding your horizons by creating something instead of playing Call of Duty 4.

I didn't expand my horizons, it was quite boring to do.
Well, I guess the grass is always greener on the other side... of Jamaica.

Sorry for not having that done already. I was a bit busy, and now I'm away. will do it tmrw.

No sweat! And thanks!
Logged

2BAM.com indie games
JamesGecko
Level 3
***



View Profile WWW
« Reply #38 on: November 29, 2009, 11:16:20 PM »

Getting all this stuff in a torrent would be pretty great. Unless we wouldn't have enough people downloading at once to make it work at a reasonable speed?
Logged
chris_b
Level 1
*


View Profile WWW
« Reply #39 on: November 30, 2009, 01:21:10 AM »

I made the Indie Asset Organizer almost exclusively for this competition.
It went through almost unnoticed but it has really useful features. It even runs on Mac.

I noticed it and would totally use it, but to be honest I was being lazy and sort of waiting until maybe someone else organized all the stuff into the correct subfolders and put the sorted files up for download.
Logged
Lon
Level 4
****



View Profile
« Reply #40 on: November 30, 2009, 01:41:07 AM »

I made the Indie Asset Organizer almost exclusively for this competition.
It went through almost unnoticed but it has really useful features. It even runs on Mac.

If there was a way to work out the assembleech to output authors and file links, leech_tool downloads everything into author-named subfolders and can be used by the Indie Asset Org. to be easily previewed (audio and graphics), drag-copied, etc.

It's a shame I spent time doing those tools and nobody is using them. Oh well.

Dude, these tools look very useful, tagging, easy copy of names, and drag and drop will be very helpful.  Assembleech looks useful too!  Need to get them files some how.  One of the reasons I have yet to use these tools is because Assemblee Part I was still going on (or at least it was), and I am not concerned about trying to collect the files till submissions are complete (wanna avoid repeatedly downloading the same, or different version of a file, can get confusing for me...).  Ill try em out shortly (hopefully all the entries will have been submitted)
Logged

“We all sorely complain of the shortness of time, and yet have much more than we know what to do with. Our lives are either spent in doing nothing at all, or in doing nothing to the purpose, or in doing nothing that we ought to do..." -Seneca
HanClinto
Level 0
***


View Profile
« Reply #41 on: November 30, 2009, 08:53:43 AM »

This is a great tool, thanks!

It looks like the list has not been refreshed in several days.  Could you please post to let us know when you update it?

Thanks!

--clint
Logged
Melly
Level 10
*****


This is how being from "da hood" is like, right?


View Profile
« Reply #42 on: November 30, 2009, 12:11:42 PM »

I plan on trying out that tool. So don't feel sad nitram. Kiss
Logged

Feel free to disregard the above.
Games: Minus / Action Escape Kitty
Sos
Level 8
***


[cluck]


View Profile WWW Email
« Reply #43 on: December 01, 2009, 12:09:08 PM »

OK, *huffpuff* I'm on and working, sorry for much of the *huff huff* delay but I got  one busy weekend and some lifeless crap-at-work to do this week (i'm a teacher, getting loaded with paperwork all of a sudden every now and then). But i'm getting too work. NOW.

btw, just updated

EDIT:


DONE! now testing nitram's leech tool

EDIT2:

IT WORKS!!!! It is a great tool, sorry for the delay in script changes too, but I was seriously out of time. I will try the IAO as soon as I lecch the contents.

EDIT3:


There is an obvious bug in my code. It doesn't actually get all the links, just some at a random basis. I will have to redo major parts of the code most probably, but that's a job for tmrw Smiley
« Last Edit: December 01, 2009, 02:45:00 PM by Sos » Logged

HanClinto
Level 0
***


View Profile
« Reply #44 on: December 01, 2009, 07:04:15 PM »

You can compile it in Mac easily with:
Quote
$ g++ -Wall leech_tool.cpp -o leech_tool
There were a couple of syntax errors, but got it to run on OS/X after inserting a couple of spaces to make the compiler happy -- it's probably just because G++ is configured by default to some ridiculously high level of paranoia.

Not sure what the problem is -- I run it, and it downloads the index file, but then after printing it to the screen, it just happily says, "Press enter to exit".  After I do so, the program exits without an error code, and there is nothing else in the directory other than the temporary cached file of links.  There are no "failed" messages -- only a dump of the temp file.

Am I doing something wrong?

I'm excited about using this tool, thanks!

--clint
Logged
Pages: 1 2 [3] 4 5
Print
Jump to:  

Theme orange-lt created by panic