如何将数据导出为 CSV 并从 Heroku 上传到单独的 FTP 站点?
我有一些数据要从数据库导出并保存为 CSV 文件,然后上传到 FTP 服务器。该网站托管在 Heroku 上,我知道您只能写入 tmp (或 log?) 目录。
这在我的本地机器上工作得很好,但在 Heroku 上不起作用。
这是我的 rake 任务:
require 'csv'
require 'net/ftp'
task :export_data => :environment do
path = "tmp/"
filename = 'test_' + Date.today.to_s + '.dat'
messages = Message.where( :foo => bar)
CSV.open(path + filename, "wb", :col_sep => '|') do |csv|
messages.each do |m|
csv << [m.id.to_s, m.name]
puts "Processing message " + m.id.to_s
end
end
puts "Uploading " + filename
ftp = Net::FTP.new('ftp.hostname.com')
ftp.login(user = "******", passwd = "*******")
ftp.puttextfile(path + filename, filename)
ftp.quit()
puts "Finished."
end
有 2 个问题:
Q1。遍历记录的速度非常慢。 5分钟200。不知道这个能不能用。
Q2。 ftp 出现错误并崩溃。它开始将文件放到 FTP 服务器上,但不向其中写入任何数据。这是一个空文件。日志显示:
Starting process with command `rake jobs:work`
2011-10-17T21:17:11+00:00 app[worker.1]: (in /app)
2011-10-17T21:17:13+00:00 heroku[worker.1]: State changed from starting to up
2011-10-17T21:17:13+00:00 app[worker.1]: rake aborted!
2011-10-17T21:17:13+00:00 app[worker.1]: Don't know how to build task 'jobs:work'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:1720:in `[]'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2040:in `invoke_task'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `block (2 levels) in top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `each'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `block in top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2058:in `standard_exception_handling'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2013:in `top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:1992:in `run'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/bin/rake:31:in `<main>'
2011-10-17T21:17:13+00:00 heroku[worker.1]: Process exited
2011-10-17T21:17:14+00:00 heroku[worker.1]: State changed from up to crashed
是否存在文件不可用的问题或文件系统的其他问题?在本地机器上工作。
有什么想法吗?
谢谢。
I have some data to export from DB and save as CSV file, and then upload to an FTP server. The site is hosted on Heroku and I understand that you can only write to the tmp (or log?) directories.
This works great on my local machine, but does not work on Heroku.
Here's my rake task:
require 'csv'
require 'net/ftp'
task :export_data => :environment do
path = "tmp/"
filename = 'test_' + Date.today.to_s + '.dat'
messages = Message.where( :foo => bar)
CSV.open(path + filename, "wb", :col_sep => '|') do |csv|
messages.each do |m|
csv << [m.id.to_s, m.name]
puts "Processing message " + m.id.to_s
end
end
puts "Uploading " + filename
ftp = Net::FTP.new('ftp.hostname.com')
ftp.login(user = "******", passwd = "*******")
ftp.puttextfile(path + filename, filename)
ftp.quit()
puts "Finished."
end
There are 2 issues:
Q1. It is incredibly slow to iterate through the records. 5 minutes for 200. I don't know if this will be usable.
Q2. The ftp has an error and crashes. It starts to put the file on the FTP server, but does not write any data to it. It is an empty file. The log reads:
Starting process with command `rake jobs:work`
2011-10-17T21:17:11+00:00 app[worker.1]: (in /app)
2011-10-17T21:17:13+00:00 heroku[worker.1]: State changed from starting to up
2011-10-17T21:17:13+00:00 app[worker.1]: rake aborted!
2011-10-17T21:17:13+00:00 app[worker.1]: Don't know how to build task 'jobs:work'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:1720:in `[]'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2040:in `invoke_task'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `block (2 levels) in top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `each'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `block in top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2058:in `standard_exception_handling'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2013:in `top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:1992:in `run'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/bin/rake:31:in `<main>'
2011-10-17T21:17:13+00:00 heroku[worker.1]: Process exited
2011-10-17T21:17:14+00:00 heroku[worker.1]: State changed from up to crashed
Is there is an issue with the file not being available or some other issue with the file system? Works on local machine.
Any ideas?
Thanks.
答案就在出于
某种原因,你的 Rakefile 不知道 jobs:work。考虑到这是一个delayed_job任务,您确定您已经正确地涵盖了delayed_job安装的每个步骤吗?
您在本地生产模式下尝试过吗?
The answer is in
For some reason, your Rakefile isn't aware of jobs:work. Considering that this is a delayed_job task, are you sure you've covered every step of the delayed_job installation properly?
Have you tried this locally in production mode?