Errors in getting one channel's data = problems

Lucas · Post by **Lucas** » Mon Jul 18, 2005 7:28 am

Occasionally, while the grabber is getting each channels data, it will have a problem pulling the file and a message pops up. At that time it retries and if file comes down, everything is OK.

If however, the retries fail as well then it skips that channels data and goes to the next channel.

In the end one can see that the grabber got data for less channels (1 or more depending on how many channel files did not come down)

The resulting xml file however shows that files that are processed after the failed channels somehow inherit the name of channel other than the correct one.

As a result, if one uses this file, the EPG is screwed up with the wrong EPG data going to the wrong channel.

Then only solution is to re-run the grabber until a totally error free run is achieved.

This is not easy though unless one sits and watches each run.
Is there away to identify the problematic run when running from a batch file and repeat the loop until it's OK?

Post by **alanbirtles** » Mon Jul 18, 2005 8:07 am

i dont really see how that happens, if the connection fails then the grabber should stop downloading, could you post the log here so i can see exactly what is happening?

Lucas · Post by **Lucas** » Mon Jul 18, 2005 9:06 pm

Well, the connection doesn't fail. The http request to fetch a channel file fails.
Then it retries 3 times and if it still can't get the file it moves to the next one.

Sometimes if the failure happens again a pop-up comes up and after pressing OK the grap stops.

I am trying to reproduce it to send the logs

Post by **alanbirtles** » Mon Jul 18, 2005 9:09 pm

after the 3 retries the grabber should always stop grabbing and not move to the next channel

Lucas · Post by **Lucas** » Mon Jul 18, 2005 9:28 pm

OK.

Here is what happens when it fails completely.
Note that in this case no XML file is produced.
Also I use a dialup 56K line.
I can cause it by accessing the web concurrently.

11:17:25 μμ - Downloading channel names/IDs
11:17:27 μμ - Processing channel names/IDs
11:17:27 μμ - Saving channels
11:17:27 μμ - Checking selected channels for deleted channels
11:17:27 μμ - Finished downloading channels

11:17:35 μμ - Grabbing 14 days of programs for 75 channels
11:19:27 μμ - Read timed out retrying 3 more times in 3 ms
11:22:52 μμ - Read timed out retrying 3 more times in 3 ms
11:23:10 μμ - Read timed out retrying 2 more times in 3 ms
11:23:13 μμ - Read timed out retrying 1 more times in 3 ms
11:23:37 μμ - Read timed out retrying 0 more times in 3 ms
11:23:37 μμ - Error downloading listings
11:23:37 μμ - Error Exception: Connection to "http://xmltv.radiotimes.com/xmltv/259.dat" failed after 3 retries

Lucas · Post by **Lucas** » Mon Jul 18, 2005 9:46 pm

Alan, I managed to also reproduce the other case where it continues.
To trigger it, as it was grabbing I openned just one web page not a series of them as above.
The result, it timed out at that moment, popped up a message, I pressed OK and it continued. In the end it generated an XML file with 74 channels rather than the 75.
One of them has wrong channel data. I have the files if you need them.

11:29:55 μμ - Grabbing 14 days of programs for 75 channels
11:30:37 μμ - Read timed out retrying 3 more times in 3 ms
11:30:48 μμ - Error EIdHTTPProtocolException: y AmiYumi~~~~~~false~false~false~false~false~false~false~false~~~Animation~Animated adventures with two cool rock stars as they travel from gig to gig or just hang out in their hometown, Tokyo.~false~25/07/2005~15:30~15:45~15

11:30:48 μμ - Error downloading listings
11:32:08 μμ - Read timed out retrying 3 more times in 3 ms
11:39:37 μμ - Read timed out retrying 3 more times in 3 ms
11:41:39 μμ - Got listings in 00:11:44
11:41:39 μμ - 74 pages requested
11:41:39 μμ - 7.875 KB downloaded
11:41:39 μμ - 24225 programs found
11:41:39 μμ - 15 days found
11:41:39 μμ - Writing XML
11:42:06 μμ - XML Written
11:42:06 μμ - Finished in 00:12:11

Lucas · Post by **Lucas** » Mon Jul 18, 2005 9:52 pm

Alan, I now enabled the log so at least I will check it after every run to ensure that it gets all channels until you find a fix.
I also disabled the MS auto-updates since they seem that be the main culprits of utilising the bandwidth.

Post by **alanbirtles** » Tue Jul 19, 2005 5:41 pm

try this temporary build http://www.birtles.org.uk/xmltv/xmltvrt2.19b.7z and let me know if it fixes the problem (it should definately stop now after a connection error). also increasing the retry delay may help reduce the connection errors

Lucas · Post by **Lucas** » Wed Jul 20, 2005 8:15 am

OK, thanks.

BTW with what util do I open the .7z file?

Post by **alanbirtles** » Wed Jul 20, 2005 9:04 am

http://www.7zip.org

Post by **alanbirtles** » Sun Jul 24, 2005 11:12 pm

has the problem reccured since you started using this new version?

Lucas · Post by **Lucas** » Mon Jul 25, 2005 8:13 am

Hi Alan,

sorry for not getting back earlier. I was away for a few days and couldn't test it until last night.

The grabber now stops properly.
I increased the retry timeout to 5000ms. Maybe more is required (10secs? or even 1min) to let the transient condition(traffic) that causes this to subside.

However, I still have to rely on manual monitoring to ensure that it gets all data and if not then re-perform the grab until it get all of them.

This is not such a big problem since even on a 56K line, 75 channels come down in approx 20mins.

But!!! I got used to every thing running automatically in the after midnight hours.

My wife now wonders what is this "new" thing that runs every week at night and keeps me from other duties........

Lucas · Post by **Lucas** » Mon Jul 25, 2005 11:05 am

Alan,

could we have the grabber return an error code <> 0 if it stops prematurely - ie not all channel data is downloaded ?

I haven't checked. Maybe it does so already.

This is so that we can loop in the .bat file until it completes with erc = 0.

Post by **alanbirtles** » Mon Jul 25, 2005 6:26 pm

try this one:
http://www.birtles.org.uk/xmltv/xmltvrt2.19b.7z
it should return 0 if the grab suceeds, 1 if the program is already running and 2 if the grab fails

Lucas · Post by **Lucas** » Tue Jul 26, 2005 9:03 am

Thanks Alan,

I 'll give it a try next couple of Days and report back.