Slow rom processing on 3.7.1

Get help with running Romcenter 3 here. Please do not post roms requests or illegal links, posts will be deleted.

Moderator: Wanderer

Forum rules
No roms requests or illegal links
danielocdh
Pong
Pong
Posts: 2
Joined: Mon Aug 12, 2013 6:40 pm

Slow rom processing on 3.7.1

Post by danielocdh »

I'm starting my mame collection, I downloaded some games and I just want to fix them with romcenter, the database and refresh processing is a bit slow but probably normal for the database size, the problem comes when I try to fix games, for example on a a relatively big 7z file(199MB), it takes 29 minutes to remove not needed roms from the file.

After a while the application becomes "not responding", fixing many games at the same time seems impossible, and with the application not responding I can't cancel when I need to or check the status. (Yesterday I left rom center working for 10 hours, until I had to shut down windows, I had to forceful close romcenter)

I have tried refresh and tools->clean database, I see no improvement

Is there any way that these processing times can be improved?

My PC specs:
windows 7 SP1 x64
AMD FX(tm)-6300 Six-Core Processor 3.50 ghz
12 gb ram
200 gb~ free space on my hard disk


Here is the log for the 29 minutes 199MB file:

Code: Select all

Ready

Fix incorrect rom files
ListView selection: 1
Rom mode : split
Bios mode : split
Sample mode : merged
Plugin : arcade.dll
Threads : 10
Green games : 27508, yellow games : 1083, red games : 100
--->Rezip: disabled
--->Test archives: disabled
--->Rename files
Done. Files renamed: 0 in 0h 0m 0s
0 threads used(max 10)
--->Fix sample path
  Only in sample split mode: skipped
Done. Samples files moved: 0 in 0h 0m 0s
--->Import missing samples (pass 1)
  No samples to fix
Done. Samples imported: 0 in 0h 0m 0s
--->Rename files
  No files to rename
Done. Files renamed: 0 in 0h 0m 0s
--->Remove roms (pass 1)
  No empty files to remove
  Delete rom roms\gamt18.7z/g18_94_5-6-0_1 (Not used)
  Delete rom roms\gamt18.7z/g18_94_5-6-0_2 (Not used)
  Create new file roms\gamt18lotc.7z for rom res01
  Move rom roms\gamt18.7z/res01 to roms\gamt18lotc.7z/res01
  Move rom roms\gamt18.7z/res02 to roms\gamt18lotc.7z/res02
  Create new file roms\gamt18a.7z for rom g18_96_5.6-0_1.bin
  Move rom roms\gamt18.7z/g18_96_5.6-0_1.bin to roms\gamt18a.7z/g18_96_5.6-0_1.bin
  Delete rom roms\gamt18.7z/g18_96_5.6-0_2.bin (Not used)
  Create new file roms\gamt18bmult.7z for rom g18_94_5.6-0_2.rom
  Move rom roms\gamt18.7z/g18_94_5.6-0_2.rom to roms\gamt18bmult.7z/g18_94_5.6-0_2.rom
  Create new file roms\gamt18d.7z for rom gamtor18_92_5.6-5#1.rom
  Move rom roms\gamt18.7z/gamtor18_92_5.6-5#1.rom to roms\gamt18d.7z/gamtor18_92_5.6-5#1.rom
  Delete rom roms\gamt18.7z/gamtor18_92_5.6-5#2.rom (Not used)
  Create new file roms\gamt18c.7z for rom gamtor18_94_5.6-5#1.rom
  Move rom roms\gamt18.7z/gamtor18_94_5.6-5#1.rom to roms\gamt18c.7z/gamtor18_94_5.6-5#1.rom
  Move rom roms\gamt18.7z/gamtor18_94_5.6-5#2.rom to roms\gamt18c.7z/gamtor18_94_5.6-5#2.rom
  Create new file roms\gamt18ent.7z for rom gr18.5_6-5.01.rom
  Move rom roms\gamt18.7z/gr18.5_6-5.01.rom to roms\gamt18ent.7z/gr18.5_6-5.01.rom
  Move rom roms\gamt18.7z/gr18.5_6-5.02.rom to roms\gamt18ent.7z/gr18.5_6-5.02.rom
  Update date and size of file roms\gamt18.7z
  Update date and size of file roms\gamt18lotc.7z
  Update date and size of file roms\gamt18a.7z
  Update date and size of file roms\gamt18bmult.7z
  Update date and size of file roms\gamt18d.7z
  Update date and size of file roms\gamt18c.7z
  Update date and size of file roms\gamt18ent.7z
Done. Roms removed: 13 in 0h 29m 12s
--->Rename roms
Done. Roms renamed: 0 in 0h 0m 0s
0 threads used(max 10)
--->Import missing roms
Done. Roms imported: 0 in 0h 0m 0s
0 threads used(max 10)
--->Create dummies
  No dummies roms to create
Done. Dummies created: 0 in 0h 0m 0s
--->Create ghost games
Done. Ghost games created: 0 in 0h 0m 0s
--->Remove roms (pass 2)
  No empty files to remove
  No roms to remove
Done. Roms removed: 0 in 0h 0m 0s
--->Remove files
  No files to remove
Done. Files removed: 0 in 0h 0m 0s
--->Import missing samples (pass 2)
  No samples to fix
--->Remove useless samples files
No samples path found.
  No samples to remove
Done. Samples deleted: 0 in 0h 0m 0s
--->Remove zip comments
Done. Zip comments removed: 0 in 0h 0m 0s
0 threads used(max 10)
Refreshing database status
Updating games status
Status updated for 47318 items.
Green games : 27508, yellow games : 1083, red games : 100
Updating fileslist
--->End fixing
----------------------------------

--->Test archives: disabled
---> 0 Samples files fixed in 0h 0m 0s
---> 0 Samples roms imported in 0h 0m 0s
---> 0 Samples files removed in 0h 0m 0s
---> 0 Samples roms removed in 0h 0m 0s
---> 0 Files renamed in 0h 0m 0s
---> 13 Roms removed in 0h 29m 12s
---> 0 Roms renamed in 0h 0m 0s
---> 0 Roms imported in 0h 0m 0s
---> 0 Dummies created in 0h 0m 0s
---> 0 Ghostgames created in 0h 0m 0s
---> 0 Files removed in 0h 0m 0s
--->Rezip: disabled
---> 0 zip comments removed in 0h 0m 0s

---> Total files fixed:    0
---> Total roms fixed:     13
---> Corrupted files:      0
---> Total Execution time: 0h 29m 15s
----------------------------------
Green games : 27508, yellow games : 1083, red games : 100
  Query Paths list
Ready

User avatar
RomCenter
Author
Author
Posts: 1519
Joined: Fri Sep 28, 2001 12:34 pm
Location: France
Contact:

Re: Slow rom processing on 3.7.1

Post by RomCenter »

Hi

This is not a normal behaviour. Maybe it is a problem with the 7zip Library. Try to convert to zip and see if it's better.
Anyway, deleting is currently quite slow. It will be improved in version 4.
Eric - RomCenter developer
Report bugs here.

danielocdh
Pong
Pong
Posts: 2
Joined: Mon Aug 12, 2013 6:40 pm

Re: Slow rom processing on 3.7.1

Post by danielocdh »

I converted all the 7z files with winrar, the behavior of romcenter on zip files was the same but with reduced times, so the not responding times were still there but because the processing was faster they lasted less. Took me like 6 hours to make all the fixes but I finished them, I forgot to mention I was using maximum compression, so maybe the speeds were ok.

Thank you.

User avatar
RomCenter
Author
Author
Posts: 1519
Joined: Fri Sep 28, 2001 12:34 pm
Location: France
Contact:

Re: Slow rom processing on 3.7.1

Post by RomCenter »

Thank you for the report.
Removing roms is not multithreaded, so this is the slowest part of operations. And this is even longer with big files (>100MB).

Version 4 will have multithreaded removing so it will hopefully greatly reduce time.
Eric - RomCenter developer
Report bugs here.

MELERIX
Street fighter II
Street fighter II
Posts: 53
Joined: Fri Feb 24, 2012 5:57 am
Location: Chile
Contact:

Re: Slow rom processing on 3.7.1

Post by MELERIX »

I have a similar issue, not during rom processing, but during importing a DB, for a example importing a MAME db from emulator could take between 15~20 minutes :?

User avatar
RomCenter
Author
Author
Posts: 1519
Joined: Fri Sep 28, 2001 12:34 pm
Location: France
Contact:

Re: Slow rom processing on 3.7.1

Post by RomCenter »

Can you identify when most time is spent ? Is it during mame extraction ?
Are you doing a fresh import or an update ?
Eric - RomCenter developer
Report bugs here.

MELERIX
Street fighter II
Street fighter II
Posts: 53
Joined: Fri Feb 24, 2012 5:57 am
Location: Chile
Contact:

Re: Slow rom processing on 3.7.1

Post by MELERIX »

fresh import

during mame's extraction (CMD window) RC log show the following (this part takes around 2-3 minutes):

Code: Select all

Extrayendo xml desde [D:\MameUI32_149u1\MAMEUI32.exe] a [C:\Users\MELERIX\AppData\Local\Temp\rc_tmpdat.dat]
Extrayendo xml desde [D:\MameUI32_149u1\MAMEUI32.exe] a [C:\Users\MELERIX\AppData\Local\Temp\rc_tmpdat.dat]
then the CMD Window is closed automatically, and RC start to taking his time with something...

RC log show the following:

Code: Select all

Rom mode : Fraccionado
Bios mode : Fraccionado
Sample mode : Fusionados
and then RC log show the following (this part takes around 1 minute):

Code: Select all

10000 lineas cargadas
20000 lineas cargadas
... etc
... etc
2950000 lineas cargadas
2960000 lineas cargadas
then RC log show the following (this part take around 13 minutes):

Code: Select all

Importar bases de datos
10000/466780 lines added
20000/466780 lines added
... etc
... etc
450000/466780 lines added
460000/466780 lines added
Done. 466780 roms added.
RC start to taking his time with something again...

then RC log show the following (this part take less than 1 minute):

Code: Select all

2963762 lineas cargadas
Rom mode : Fraccionado
Bios mode : Fraccionado
Sample mode : Fusionados
Reorganizando.
then RC log show the following:

Code: Select all

transferir 1/5
Tiempo de ejecución: 0h 1m 36s
transferir 2/5
Tiempo de ejecución: 0h 0m 49s
transferir 3/5
Tiempo de ejecución: 0h 5m 2s
transferir 4/5
Tiempo de ejecución: 0h 3m 45s
transferir 5/5
then RC log show the following (this part take around 3 minutes):

Code: Select all

Remover archivos temporales
Se esta reconstruyendo y reorganizando la base de datos. Por favor espere...
and finally RC log show the following:

Code: Select all

Tiempo de ejecución: 0h 38m 28s
Cerrar C:\Users\Public\Documents\romcenter\Datas\MAMEUI32 v0.149u1.rdt
Abrir C:\Users\Public\Documents\romcenter\Datas\MAMEUI32 v0.149u1.rdt
Cargando la base de datos C:\Users\Public\Documents\romcenter\Datas\MAMEUI32 v0.149u1.rdt
Versión de la base de datos MAMEUI32 v0.149u1  = 3,7002
Rom mode : Fraccionado
Bios mode : Fraccionado
Sample mode : Fusionados
Plugin : arcade.dll
Cargando la lista de juegos
28691 juegos cargados.
Actualizando el estado de los juegos
Estado actualizado para 47318 objetos.
Juegos correctos (verde) : 4, Juegos para arreglar (amarillo) : 46, juegos dañados o faltantes (rojo) : 28641
Actualizando la lista de archivos
  Solicitando la lista de rutas.
¡Listo para comenzar!
thats all.

sarasmith
Pong
Pong
Posts: 2
Joined: Tue Dec 09, 2014 4:33 pm

Re: Slow rom processing on 3.7.1

Post by sarasmith »

I'm encountering what seems to be a bug with the vector games. It is labeling all the unique roms in each clone as "not needed in this file", and the shared roms which are all in the vector.
sara

Locked