如何将数据导出为 CSV 并从 Heroku 上传到单独的 FTP 站点?

发布于 12-10 10:52 字数 2345 浏览 1 评论 0原文

我有一些数据要从数据库导出并保存为 CSV 文件,然后上传到 FTP 服务器。该网站托管在 Heroku 上,我知道您只能写入 tmp (或 log?) 目录。

这在我的本地机器上工作得很好,但在 Heroku 上不起作用。

这是我的 rake 任务:

require 'csv' 
require 'net/ftp'

task :export_data => :environment do
  path = "tmp/" 
  filename = 'test_' + Date.today.to_s + '.dat' 

  messages = Message.where( :foo => bar) 
  CSV.open(path + filename, "wb", :col_sep => '|') do |csv| 

    messages.each do |m| 
      csv << [m.id.to_s, m.name] 
      puts "Processing message " + m.id.to_s 
    end 
  end 

  puts "Uploading " + filename 
  ftp = Net::FTP.new('ftp.hostname.com') 
  ftp.login(user = "******", passwd = "*******") 
  ftp.puttextfile(path + filename, filename) 
  ftp.quit() 

  puts "Finished." 
end

有 2 个问题:

Q1。遍历记录的速度非常慢。 5分钟200。不知道这个能不能用。

Q2。 ftp 出现错误并崩溃。它开始将文件放到 FTP 服务器上,但不向其中写入任何数据。这是一个空文件。日志显示:

Starting process with command `rake jobs:work`
2011-10-17T21:17:11+00:00 app[worker.1]: (in /app)
2011-10-17T21:17:13+00:00 heroku[worker.1]: State changed from starting to up
2011-10-17T21:17:13+00:00 app[worker.1]: rake aborted!
2011-10-17T21:17:13+00:00 app[worker.1]: Don't know how to build task 'jobs:work'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:1720:in `[]'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2040:in `invoke_task'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `block (2 levels) in top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `each'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `block in top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2058:in `standard_exception_handling'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2013:in `top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:1992:in `run'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/bin/rake:31:in `<main>'
2011-10-17T21:17:13+00:00 heroku[worker.1]: Process exited
2011-10-17T21:17:14+00:00 heroku[worker.1]: State changed from up to crashed

是否存在文件不可用的问题或文件系统的其他问题?在本地机器上工作。

有什么想法吗?

谢谢。

I have some data to export from DB and save as CSV file, and then upload to an FTP server. The site is hosted on Heroku and I understand that you can only write to the tmp (or log?) directories.

This works great on my local machine, but does not work on Heroku.

Here's my rake task:

require 'csv' 
require 'net/ftp'

task :export_data => :environment do
  path = "tmp/" 
  filename = 'test_' + Date.today.to_s + '.dat' 

  messages = Message.where( :foo => bar) 
  CSV.open(path + filename, "wb", :col_sep => '|') do |csv| 

    messages.each do |m| 
      csv << [m.id.to_s, m.name] 
      puts "Processing message " + m.id.to_s 
    end 
  end 

  puts "Uploading " + filename 
  ftp = Net::FTP.new('ftp.hostname.com') 
  ftp.login(user = "******", passwd = "*******") 
  ftp.puttextfile(path + filename, filename) 
  ftp.quit() 

  puts "Finished." 
end

There are 2 issues:

Q1. It is incredibly slow to iterate through the records. 5 minutes for 200. I don't know if this will be usable.

Q2. The ftp has an error and crashes. It starts to put the file on the FTP server, but does not write any data to it. It is an empty file. The log reads:

Starting process with command `rake jobs:work`
2011-10-17T21:17:11+00:00 app[worker.1]: (in /app)
2011-10-17T21:17:13+00:00 heroku[worker.1]: State changed from starting to up
2011-10-17T21:17:13+00:00 app[worker.1]: rake aborted!
2011-10-17T21:17:13+00:00 app[worker.1]: Don't know how to build task 'jobs:work'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:1720:in `[]'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2040:in `invoke_task'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `block (2 levels) in top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `each'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2019:in `block in top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2058:in `standard_exception_handling'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:2013:in `top_level'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/lib/ruby/1.9.1/rake.rb:1992:in `run'
2011-10-17T21:17:13+00:00 app[worker.1]: /usr/ruby1.9.2/bin/rake:31:in `<main>'
2011-10-17T21:17:13+00:00 heroku[worker.1]: Process exited
2011-10-17T21:17:14+00:00 heroku[worker.1]: State changed from up to crashed

Is there is an issue with the file not being available or some other issue with the file system? Works on local machine.

Any ideas?

Thanks.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

多孤肩上扛2024-12-17 10:52:47

答案就在出于

2011-10-17T21:17:13+00:00 app[worker.1]: Don't know how to build task 'jobs:work'

某种原因,你的 Rakefile 不知道 jobs:work。考虑到这是一个delayed_job任务,您确定您已经正确地涵盖了delayed_job安装的每个步骤吗?

您在本地生产模式下尝试过吗?

The answer is in

2011-10-17T21:17:13+00:00 app[worker.1]: Don't know how to build task 'jobs:work'

For some reason, your Rakefile isn't aware of jobs:work. Considering that this is a delayed_job task, are you sure you've covered every step of the delayed_job installation properly?

Have you tried this locally in production mode?

梦年海沫深2024-12-17 10:52:47

回答:

Q1:不慢。它看起来很慢,因为慢的部分正在输出到控制台。

Q2:通过设置 ftp.passive = true 修复了 FTP

Answers:

Q1: It's not slow. It only looks slow because the slow part is outputting to the console.

Q2: FTP was fixed by setting ftp.passive = true

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文