Colors of Ray+Hue'

while read a를 이용한 화면출력 및 저장 및 파일 만들기

shell2015. 10. 14. 06:45

cat test2.sh | while read a ; do echo "${a}"; echo "${a}" >> logfile.txt; done

ls * | while read a; do cat $a; done


'shell' 카테고리의 다른 글

scale bc 사용법  (0) 2016.04.20
time -o (--output=) 사용법 in Ubuntu 12.04  (0) 2016.03.19
[shell] find 사용법  (0) 2015.11.03

PySpark Import Error

spark2015. 10. 14. 06:11 
from pyspark import SparkContext
ImportError: No module named py4j.java_gateway
echo $SPARK_HOME 을 확인할것, 만일 spark 설치 폴더가 아닐경우, export 할것

해당 라이브러리를 사용하고자 할경우, 2개의 PATH (SPARK_HOME, PYTHONPATH) 를 설정해야 한다. 


export SPARK_HOME=/working/spark/spark-1.3.1-bin-hadoop2.6

export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH

py4j 를 사용하기 위해서 해당 디렉토리의 zip 파일을 path로 설정해야 한다. 


.bashrc에 포함하여 해당 account의 shell이 실행될 때마다 위의 path export가 실행 될 수 있도록 한다. 


  


'spark' 카테고리의 다른 글

spark RDD  (0) 2015.11.20
spark source code tag cscope building  (0) 2015.11.20
Spark Installation on ubuntu 12.04  (0) 2015.10.14

Spark Installation on ubuntu 12.04

spark2015. 10. 14. 06:03

1. 소스를 가져온다. spark-1.4.1

 http://spark.apache.org/downloads.html 

2. 압축풀고 메이븐 실행 

 mvn -DskipTests clean package

3. (2)가 실패할 경우, 메이븐(maven) 설치

    3.1. maven 3.3 source 다운로드 (http://maven.apache.org/install.html)

    3.2. 압축풀고, maven 3.3 directory에 bin 폴더를 생성

    3.3. 자바 홈 디렉토리의 환경변수가 제대로 설정되어 있는지 확인

 echo $JAVA_HOME

        3.3.1 아닐경우, export JAVA_HOME 설정 (update-alternatives 정보 확인) 

      /usr/lib/jvm/jdk1.8.0_11 (최소 jde 7 이상)

    3.4. 환경변수 설정

 export PATH=/opt/apache-maven-3.3.3/bin:$PATH

4. (2) 재실행 --> 설치완료


'spark' 카테고리의 다른 글

spark RDD  (0) 2015.11.20
spark source code tag cscope building  (0) 2015.11.20
PySpark Import Error  (0) 2015.10.14

etags

Linux Kernel2015. 7. 21. 08:53

etags

find `pwd` -name "*.cc" -o -name "*.[cChH]" -o -name "*.sh" -o -name "*.cpp" -o -name "*.md" -print | xargs etags -a

'Linux Kernel' 카테고리의 다른 글

LXC 관련 자료  (0) 2015.10.17
ZEST [thezest] 사용법  (0) 2015.10.16
qemu 설치및 사용  (0) 2015.10.16
permanent 환경변수 설정 on ubuntu 12.04  (0) 2015.10.14
ldconfig deferred processing now taking place?  (0) 2015.07.18

examplie: test application for rocksdb compile

Dababase2015. 7. 18. 10:05

g++ -g -Wall -Werror -I. -I./include -std=c++11  -DROCKSDB_PLATFORM_POSIX  -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_ATOMIC_PRESENT -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=google -DZLIB -DBZIP2   -DHAVE_JEMALLOC -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -c db/db_bench.cc -o db/db_bench.o 


g++ db/db_bench.o db/builder.o db/c.o db/column_family.o db/compaction.o db/compaction_picker.o db/db_filesnapshot.o db/dbformat.o db/db_impl.o db/db_impl_debug.o db/db_impl_readonly.o db/db_iter.o db/db_stats_logger.o db/file_indexer.o db/filename.o db/internal_stats.o db/log_reader.o db/log_writer.o db/memtable.o db/memtable_list.o db/merge_helper.o db/merge_operator.o db/repair.o db/table_cache.o db/table_properties_collector.o db/tailing_iter.o db/transaction_log_impl.o db/version_edit.o db/version_set.o db/write_batch.o table/block_based_table_builder.o table/block_based_table_factory.o table/block_based_table_reader.o table/block_builder.o table/block.o table/block_hash_index.o table/filter_block.o table/flush_block_policy.o table/format.o table/iterator.o table/merger.o table/meta_blocks.o table/plain_table_builder.o table/plain_table_factory.o table/plain_table_reader.o table/table_properties.o table/two_level_iterator.o util/arena.o util/auto_roll_logger.o util/blob_store.o util/bloom.o util/build_version.o util/cache.o util/coding.o util/comparator.o util/crc32c.o util/dynamic_bloom.o util/env.o util/env_hdfs.o util/env_posix.o util/filter_policy.o util/hash.o util/hash_cuckoo_rep.o util/hash_linklist_rep.o util/hash_skiplist_rep.o util/histogram.o utilities/backupable/backupable_db.o utilities/geodb/geodb_impl.o utilities/merge_operators/put.o utilities/merge_operators/string_append/stringappend2.o utilities/merge_operators/string_append/stringappend.o utilities/merge_operators/uint64add.o utilities/redis/redis_lists.o utilities/ttl/db_ttl_impl.o util/ldb_cmd.o util/ldb_tool.o util/log_buffer.o util/logging.o util/murmurhash.o util/options.o util/perf_context.o util/skiplistrep.o util/slice.o util/statistics.o util/status.o util/string_util.o util/sync_point.o util/thread_local.o util/vectorrep.o util/xxhash.o port/port_posix.o port/stack_trace.o  ./util/testutil.o   -o db_bench  -lpthread -lrt -lsnappy -lgflags -lz -lbz2 

'Dababase' 카테고리의 다른 글

postgresql 9.3 on SpecVirt 작업 일지  (0) 2016.02.05
port forwarding  (0) 2016.02.02
Rocks DB 설치 on Ubuntu 12.04 LTS  (0) 2015.07.18
Installing gflags 12.04  (0) 2015.07.17
RocksDB Installation on Ubuntu 12.04  (0) 2015.07.17

ldconfig deferred processing now taking place?

Linux Kernel2015. 7. 18. 09:37

This is normal and to do with triggers, if a package requires ldconfig to be run after installing a library then the trigger cause the command to be run only once at the end of installation rather than after every library is installed. ldconfig creates the necessary links and cache to shared libraries.

'Linux Kernel' 카테고리의 다른 글

LXC 관련 자료  (0) 2015.10.17
ZEST [thezest] 사용법  (0) 2015.10.16
qemu 설치및 사용  (0) 2015.10.16
permanent 환경변수 설정 on ubuntu 12.04  (0) 2015.10.14
etags  (0) 2015.07.21

Rocks DB 설치 on Ubuntu 12.04 LTS

Dababase2015. 7. 18. 09:18

How to install RocksDB on Ubuntu 12.04 LTS

Posted: Tháng Mười Hai 4, 2013 in Database
Tags: , , , , , , 
1

If you want to know more information about RocksDB, go to there: http://rocksdb.org/

RockDB requires some new libraries that do not support ubuntu 12.04. I’ll help you install step by step to do it.

1) Install gcc and g++ version  4.8

– Check your gcc and g++ version:

1
gcc --version

and

1
g++ --version

If gcc and g++ haven’t installed in your OS or their version < 4.7, you must install them by comand lines below:

– Add PPA to your repositories:

1
2
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update

– Unistall the alternative:

1
2
sudo update-alternatives --remove-all gcc
sudo update-alternatives --remove-all g++

– Install:

1
2
sudo apt-get install gcc-4.8
sudo apt-get install g++-4.8

– Alternative packages install:

1
2
3
4
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 20
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 20
sudo update-alternatives --config gcc
sudo update-alternatives --config g++

at the end:

1
2
3
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade

2) Install the libararies:

* [zlib](http://www.zlib.net/) – a library for data compression.

1
sudo apt-get install zlib1g-dev

* [bzip2](http://www.bzip.org/) – a library for data compression.

1
sudo apt-get install libbz2-dev

* [snappy](https://code.google.com/p/snappy/) – a library for fast

     data compression.

1
sudo apt-get install libsnappy-dev

* [gflags](https://code.google.com/p/gflags/) – a library that handles

– Go to page: https://github.com/schuhschuh/gflags

– Download gflags compressed file.

– Extract file and cd to gflags-master folder

1
./configure
1
make
1
sudo make install

3) Install rocksdb:

– Download zip file from: https://github.com/facebook/rocksdb

– Extract file and cd to rocksdb-master

1
make
1
sudo make install

4) Create test project

After installing all libraries required by RocksDB. We will create a test project. In this case, I use Netbeans IDE:

1. Create an C++ Application

2. Create library

– Go to your project folder, in this project folder, we create a folder and named it as “lib”.

– Go to folder rocksdb-master that we had built and copy folder rocksdb-master/include/rocksdb to <your project folder>/lib.

– Copy library rocks-master/librocksdb.a to <your project folder>/lib and /usr/lib.

– Now make sure that you had installed bz2 and snappy. Go to /usr/lib and copy libraries: libbz2.a, libsnappy.a to <your project folder>/lib.

3. Configure your project:

Right click on Netbeans project and chose Properties.

– In Build > C++Complier > C++ Standard  chose C++11.

– In Build > C++Complier> Include Directories and Headers, link to <your project folder>/lib.

– In Build > Linker > Addition Library Directories, link to <your project folder>/lib.

– In Build > Linker > Libraries, chose Add Libraries and chose all .a files in <your project folder>/lib we had copied.

– In Build > Linker > Additional Options: add “-lz -lpthread -lrt”.

Now let’s try to copy source code below to main file in your project and build:

1
2
3
4
5
6
7
8
9
10
11
#include <assert.h>
#include "rocksdb/db.h"
 
int main(int argc, char** argv) {
 rocksdb::DB *db;
 rocksdb::Options options;
 options.create_if_missing = true;
 rocksdb::Status status = rocksdb::DB::Open(options, "/tmp/testdb/", &db);
 assert(status.ok());
 return 0;
}

Enjoy!

original source: https://bienuit.wordpress.com/2013/12/04/how-to-install-rocksdb-on-ubuntu-12-04-lts/

'Dababase' 카테고리의 다른 글

postgresql 9.3 on SpecVirt 작업 일지  (0) 2016.02.05
port forwarding  (0) 2016.02.02
examplie: test application for rocksdb compile  (0) 2015.07.18
Installing gflags 12.04  (0) 2015.07.17
RocksDB Installation on Ubuntu 12.04  (0) 2015.07.17

Installing gflags 12.04

Dababase2015. 7. 17. 11:51

Installing gflags 12.04

Manual package build (backport)

This is a very very verbose description - for anyone building a package for the first time.

1.    Install basic packages to build software and packages: build-essential   and devscripts  .

2.    Go to the source package (gflags) page at Launchpad:https://launchpad.net/ubuntu/+source/gflags

3.    Unfold the section for "The Quantal Quetzal (supported) 2.0-1" version.

4.    Locate the source package description file (.dsc extension). At the time of writing this ishttps://launchpad.net/ubuntu/+archive/primary/+files/gflags_2.0-1.dsc

5.    Copy the link to your clipboard.

6.    Open a terminal and download the source package using dget:

7.  dget https://launchpad.net/ubuntu/+archive/primary/+files/gflags_2.0-1.dsc

This will fail the first time:

gpg: Signature made Thu 31 May 2012 14:48:41 CEST using RSA key ID 8AE09345

gpg: Can't check signature: public key not found

Validation FAILED!!

8.    Install the required RSA key as in the error message above:

9.  gpg --keyserver keyserver.ubuntu.com --recv-key 8AE09345

10.  Configure the DPKG development scripts to use your GPG keyring:

11. echo 'DSCVERIFY_KEYRINGS="/etc/apt/trusted.gpg:~/.gnupg/pubring.gpg"' > ~/.devscripts

See Added key, but dget still shows "gpg: Can't check signature: public key not found" for why.

12.  Run the earlier dget command again. This should now succeed.

13.  Hop into the directory created:

14. cd gflags-2.0/

15.  Try building the package.

16. debuild -uc -us

Explanation for the options: unsigned changes file, unsigned new .dsc file. As you are not redistributing the package, there's no need to sign anything.

This may fail due to missing build dependencies, e.g.:

dpkg-checkbuilddeps: Unmet build dependencies: debhelper

Note this is really system specific.

17.  Install the build dependencies (satisfy all above from the output you get), e.g.:

18. sudo apt-get install debhelper

19.  Try building the package again:

20. debuild -uc -us

21.  One directory below, you'll find your packages, e.g.:

22. $ cd ..

23. $ ls -al *gflags*.deb

24. -rw-r--r-- 1 gert gert 108450 Jun 24 18:59 libgflags2_2.0-1_amd64.deb

25. -rw-r--r-- 1 gert gert 147590 Jun 24 18:59 libgflags-dev_2.0-1_amd64.deb

26. -rw-r--r-- 1 gert gert  14778 Jun 24 18:59 libgflags-doc_2.0-1_all.deb

27.  Install them:

28. sudo dpkg -i *gflags*.deb

In case this fails because of binary dependencies not satisfied, run

sudo apt-get install -f

Done!

Original Source: http://askubuntu.com/questions/312173/installing-gflags-12-04

 

'Dababase' 카테고리의 다른 글

postgresql 9.3 on SpecVirt 작업 일지  (0) 2016.02.05
port forwarding  (0) 2016.02.02
examplie: test application for rocksdb compile  (0) 2015.07.18
Rocks DB 설치 on Ubuntu 12.04 LTS  (0) 2015.07.18
RocksDB Installation on Ubuntu 12.04  (0) 2015.07.17

RocksDB Installation on Ubuntu 12.04

Dababase2015. 7. 17. 11:47

RocksDB Installation on Ubuntu 12.04

-          Platform Setup

1.       Gcc 4.7 to support C++10

                                                               i.      Check gcc version, if it is less than 4.7,

1.       sudo dpkg -l | grep gcc | awk '{print $2}'

                                                             ii.      Install gcc and g++ 4.7

1.       sudo add-apt-repository ppa:ubuntu-toolchain-r/test

2.       sudo apt-get update

3.       sudo apt-get install gcc-4.7 g++-4.7

                                                            iii.      alternative setup

1.       $ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40

2.       $ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 20

3.       $ sudo update-alternatives --display gcc

4.       If the gcc 4.7 is not setup yet, update it manually

a.       $sudo update-alternatives –config gcc

b.      Choose gcc 4.7

5.       $ gcc ---version

                                                           iv.      G++-4.7, repeat the process (iii)

                                                             v.      Install zlib

1.       $sudo apt-get install zlib1g-dev

                                                           vi.      Install snappy

1.       $sudo apt-get install libsnappy-dev

2.       [may not be required] $sudo apt-get install snappy

                                                          vii.      Install bzip2

1.       $sudo apt-get install libbz2-dev

                                                        viii.      Install gflags

1.       For static library, and all our tools and unit tests

2.       Refer the Installation Guide, gflags on Ubuntu 12.04 

-          Installation RocksDB

1.       Download RocksDB

a.       wget  https://github.com/facebook/rocksdb/archive/master.zip

2.       Compile

a.       make static_lib --> make (it compiles db_bench and other utilities for test)

b.      make check

c.       make all


'Dababase' 카테고리의 다른 글

postgresql 9.3 on SpecVirt 작업 일지  (0) 2016.02.05
port forwarding  (0) 2016.02.02
examplie: test application for rocksdb compile  (0) 2015.07.18
Rocks DB 설치 on Ubuntu 12.04 LTS  (0) 2015.07.18
Installing gflags 12.04  (0) 2015.07.17

티스토리툴바