top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

How to write fast into a file in python?

+1 vote
537 views

I need to write numbers into a file upto 50mb and it should be fast can any one help me how to do that? I had written the following code..

def create_file_numbers_old(filename, size):
start = time.clock()

value = 0
with open(filename, "w") as f:
while f.tell()< size:
f.write("{0}n".format(value))
value += 1

end = time.clock()

print "time taken to write a file of size", size, " is ", (end -start), "seconds n"

it takes about 20sec i need 5 to 10 times less than that.

posted May 17, 2013 by anonymous

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button

2 Answers

+1 vote
 
Best answer

20 seconds to write how many numbers? If you are doing

create_file_numbers_old(filename, 5)

then 20 seconds is really slow. But if you are doing:

create_file_numbers_old(filename, 50000000000000)

then 20 seconds is amazingly fast.

Try this instead, it may be a little faster:

def create_file_numbers_old(filename, size):
 count = value = 0
 with open(filename, 'w') as f:
 while count < size:
 s = '%dn' % value
 f.write(s)
 count += len(s)
 value += 1

If this is still too slow, you can try three other tactics:

1) pre-calculate the largest integer that will fit in size bytes, then use a for-loop instead of a while loop:

 maxn = calculation(...)
 with open(filename, 'w') as f:
 for i in xrange(maxn):
 f.write('%dn' % i)

2) Write an extension module in C that writes to the file.
3) Get a faster hard drive, and avoid writing over a network.

answer May 17, 2013 by anonymous
0 votes

Most of the time is spent figuring out whether the file has reached its limit size. If you want Python to go fast, just specify the data. On my Linux system, it takes 11 seconds to write the first 6338888 values,
which is just under 50mb. If I write the obvious loop, writing that many values takes .25 seconds.

answer May 17, 2013 by anonymous
Similar Questions
0 votes

I need to write into a file for a project which will be evaluated on the basis of time. What is the fastest way to write 200 Mb of data, accumulated as a list into a file.

Presently I am using this:

with open('index.txt','w') as f:
 f.write("".join(data))
 f.close()

where data is a list of strings which I want to dump into the index.txt file

+1 vote

When I input usb line with my android phone into the pc , there are two disks j: and k: (type :removable disk) displayed in win7.

I can get my android phone bluetooth mac address .

import bluetooth
nearby_devices = bluetooth.discover_devices(lookup_names = True)
for addr, phoneName in nearby_devices:
 print(addr)

it is XX:XX:XX:XX:XX:XX

Now how can I write a file into the disk j: of my android phone(bluetooth mac is XX:XX:XX:XX:XX:XX )?

0 votes

I'm trying to write a python script that writes some content to a file root through sudo, but it's not working at all. I am using:

 channel = 'stable'
 config_file = '/opt/ldg/etc/channel.conf'
 command = ['echo', '-n', channel, '|', 'sudo', 'tee', config_file]
 p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 out, _ = p.communicate()

But it seems as if this isn't doing anything.

I just want to write the contents of the variable channel to the file /opt/ldg/etc/channel.conf. But whatever I try just doesn't work. Can anyone offer any pointers?

+2 votes

Basically, is it possible to compile multiple unrelated python scripts into a single exe file, so when execute it several python programs are run at once.

In order to use this on another machine without python installed and only by using one single file.

+1 vote

With Mozrepl addon in Firefox and Python I do:

> import telnetlib
> tn = telnetlib.Telnet(r'127.0.0.1', 4242, 5)
> tn.read_eager()
'nWelcome to MozRepl.nn - If you get stuck at the "'
> tn.read_until("repl> ")
...snip...
> tn.write(r'alert(window.content.location.href)'+"n")

and I get an alert box with the URL of the active tab.

But how do I read that URL into a python variable? Something like tn.write(r';var zz = window.content.location.href'+ "n") but that doesn't get it into python.

...